2024-04-24T15:38:25.478 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T15:38:25.478 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585 --description rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd} --name yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/orig.config.yaml 2024-04-24T15:38:25.514 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T15:38:25.600 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585 branch: wip-yuriw4-testing-20240423.151325-quincy description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd} email: yweinste@redhat.com first_in_suite: false job_id: '7671585' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuriw4-testing-20240423.151325-quincy ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw4-testing-20240423.151325-quincy sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 9954 sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuriw4-testing-20240423.151325-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbjG2d46Cc6vc2UmtV449lwgybDTTObm0+JMiC2GMj1r1w+0bfO0hVfPSwbiPlEWN/cYdSTL5QgYOrgnfVOpGM= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM/i7Hr+2nm2cYmKiAcPHe5wxWBQwHIvdGs+S4GkzEJo/mo+rFUjEgQ5EC9apO6MB8RPlPugryuaoONVEAC2L5U= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaKRPvBPlQ9VkPpvDNyl6FNgnrz5pOCXNsDaKxJDM5HvfdIQlDprbmi1ENIFrhnMHIVeBZecsWWr4ayG32MqAo= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_14:09:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T15:38:25.600 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa; will attempt to use it 2024-04-24T15:38:25.601 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa/tasks 2024-04-24T15:38:25.601 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T15:38:25.602 INFO:teuthology.task.internal:Checking packages... 2024-04-24T15:38:25.625 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '330067dbce47655d1f762ef14d546d2fde16a6a9' 2024-04-24T15:38:25.625 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T15:38:25.625 INFO:teuthology.packaging:ref: None 2024-04-24T15:38:25.625 INFO:teuthology.packaging:tag: None 2024-04-24T15:38:25.625 INFO:teuthology.packaging:branch: wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T15:38:25.625 INFO:teuthology.packaging:sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:38:25.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T15:38:25.837 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-915.g330067db 2024-04-24T15:38:25.839 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T15:38:25.845 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T15:38:25.845 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T15:38:25.859 INFO:teuthology.task.internal:Saving configuration 2024-04-24T15:38:25.872 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T15:38:25.879 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T15:38:25.901 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 15:33:19.975752', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbjG2d46Cc6vc2UmtV449lwgybDTTObm0+JMiC2GMj1r1w+0bfO0hVfPSwbiPlEWN/cYdSTL5QgYOrgnfVOpGM='} 2024-04-24T15:38:25.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 15:33:19.977844', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM/i7Hr+2nm2cYmKiAcPHe5wxWBQwHIvdGs+S4GkzEJo/mo+rFUjEgQ5EC9apO6MB8RPlPugryuaoONVEAC2L5U='} 2024-04-24T15:38:25.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 15:33:19.976933', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaKRPvBPlQ9VkPpvDNyl6FNgnrz5pOCXNsDaKxJDM5HvfdIQlDprbmi1ENIFrhnMHIVeBZecsWWr4ayG32MqAo='} 2024-04-24T15:38:25.939 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T15:38:25.945 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-24T15:38:25.945 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-04-24T15:38:25.945 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['client.0'] 2024-04-24T15:38:25.945 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T15:38:26.047 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f75a9401f70>, signals=[15]) 2024-04-24T15:38:26.048 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T15:38:26.054 INFO:teuthology.task.internal:Opening connections... 2024-04-24T15:38:26.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T15:38:26.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:38:26.144 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T15:38:26.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:38:26.218 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-24T15:38:26.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:38:26.286 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T15:38:26.293 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-24T15:38:26.315 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-24T15:38:26.315 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8.6 (Ootpa)" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8.6" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2024-04-24T15:38:26.376 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-24T15:38:26.377 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-24T15:38:26.378 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-24T15:38:26.400 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-04-24T15:38:26.420 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-04-24T15:38:26.421 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.6 (Ootpa)" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.6" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2024-04-24T15:38:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-24T15:38:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-24T15:38:26.481 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-04-24T15:38:26.500 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-24T15:38:26.521 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-24T15:38:26.521 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux" 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)" 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel" 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora" 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6" 2024-04-24T15:38:26.581 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-24T15:38:26.582 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-24T15:38:26.582 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-24T15:38:26.600 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T15:38:26.608 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T15:38:26.614 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T15:38:26.614 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-24T15:38:26.616 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-04-24T15:38:26.618 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-24T15:38:26.640 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T15:38:26.646 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T15:38:26.646 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-24T15:38:26.677 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-04-24T15:38:26.679 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-24T15:38:26.708 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T15:38:26.784 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T15:38:26.799 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T15:38:26.799 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T15:38:26.799 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T15:38:26.800 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T15:38:26.800 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T15:38:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:26.819 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-24T15:38:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:26.820 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-24T15:38:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:26.821 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-24T15:38:26.878 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.878 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-04-24T15:38:26.880 INFO:teuthology.orchestra.run.smithi122.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.880 INFO:teuthology.task.kernel:Running kernel on smithi122: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-04-24T15:38:26.881 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.882 INFO:teuthology.task.kernel:Running kernel on smithi032: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:26.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-04-24T15:38:28.343 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:28.343 INFO:teuthology.orchestra.run.smithi122.stdout:Unable to read consumer identity 2024-04-24T15:38:28.538 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:28.538 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2024-04-24T15:38:28.556 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:28.556 INFO:teuthology.orchestra.run.smithi032.stdout:Unable to read consumer identity 2024-04-24T15:38:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:28.590 INFO:teuthology.orchestra.run.smithi122.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:28.871 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:28.880 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:30.364 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-24T15:38:30.751 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-24T15:38:30.757 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-04-24T15:38:31.192 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2024-04-24T15:38:33.801 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 16 MB 00:03 2024-04-24T15:38:34.840 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 534 kB/s | 24 kB 00:00 2024-04-24T15:38:35.534 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-24T15:38:35.535 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-24T15:38:35.606 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:38:35.608 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:38:35.608 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:38:35.749 DEBUG:teuthology.orchestra.run.smithi122:> echo no | sudo yum reinstall kernel || true 2024-04-24T15:38:36.086 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:36.086 INFO:teuthology.orchestra.run.smithi122.stdout:Unable to read consumer identity 2024-04-24T15:38:36.325 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:36.325 INFO:teuthology.orchestra.run.smithi122.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:36.325 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:36.636 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 03:38:34 PM UTC. 2024-04-24T15:38:36.704 INFO:teuthology.orchestra.run.smithi122.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:36.712 INFO:teuthology.orchestra.run.smithi122.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:36.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum reinstall -y kernel || true 2024-04-24T15:38:37.126 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:37.126 INFO:teuthology.orchestra.run.smithi122.stdout:Unable to read consumer identity 2024-04-24T15:38:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:37.378 INFO:teuthology.orchestra.run.smithi122.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:38:37.435 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 399 kB/s | 24 kB 00:00 2024-04-24T15:38:37.516 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 16 MB 00:06 2024-04-24T15:38:37.681 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 03:38:34 PM UTC. 2024-04-24T15:38:37.744 INFO:teuthology.orchestra.run.smithi122.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:37.744 INFO:teuthology.orchestra.run.smithi122.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:37.771 INFO:teuthology.orchestra.run.smithi122.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:37.817 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T15:38:37.981 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:37.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:37.982 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T15:38:37.982 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-04-24T15:38:37.982 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T15:38:38.017 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-24T15:38:38.030 DEBUG:teuthology.parallel:result is None 2024-04-24T15:38:38.170 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-24T15:38:38.171 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-24T15:38:38.263 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:38:38.264 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:38:38.264 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:38:38.410 DEBUG:teuthology.orchestra.run.smithi086:> echo no | sudo yum reinstall kernel || true 2024-04-24T15:38:38.785 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:38.785 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2024-04-24T15:38:39.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:39.015 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:39.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:39.326 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 03:38:37 PM UTC. 2024-04-24T15:38:39.398 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:39.403 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:39.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum reinstall -y kernel || true 2024-04-24T15:38:39.840 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:39.840 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2024-04-24T15:38:40.062 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:40.062 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:40.063 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:38:40.363 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 03:38:37 PM UTC. 2024-04-24T15:38:40.426 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:40.426 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:40.454 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:40.500 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T15:38:40.661 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:40.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:40.662 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T15:38:40.662 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T15:38:40.662 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T15:38:40.698 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-04-24T15:38:40.713 DEBUG:teuthology.parallel:result is None 2024-04-24T15:38:41.549 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 460 kB/s | 24 kB 00:00 2024-04-24T15:38:42.377 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-24T15:38:42.377 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-24T15:38:42.464 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:38:42.466 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:38:42.466 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:38:42.604 DEBUG:teuthology.orchestra.run.smithi032:> echo no | sudo yum reinstall kernel || true 2024-04-24T15:38:42.990 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:42.990 INFO:teuthology.orchestra.run.smithi032.stdout:Unable to read consumer identity 2024-04-24T15:38:43.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:43.208 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:43.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:43.577 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 03:38:41 PM UTC. 2024-04-24T15:38:43.662 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:43.662 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:43.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum reinstall -y kernel || true 2024-04-24T15:38:44.144 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:38:44.144 INFO:teuthology.orchestra.run.smithi032.stdout:Unable to read consumer identity 2024-04-24T15:38:44.351 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:44.352 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:38:44.352 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:38:44.712 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 03:38:41 PM UTC. 2024-04-24T15:38:44.788 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:38:44.788 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:38:44.821 INFO:teuthology.orchestra.run.smithi032.stderr:Error: No packages marked for reinstall. 2024-04-24T15:38:44.875 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T15:38:45.057 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:45.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:38:45.057 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T15:38:45.057 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T15:38:45.057 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T15:38:45.095 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-04-24T15:38:45.108 DEBUG:teuthology.parallel:result is None 2024-04-24T15:38:45.108 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T15:38:45.115 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T15:38:45.116 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T15:38:45.151 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T15:38:45.154 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T15:38:45.175 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T15:38:45.181 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T15:38:45.187 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T15:38:45.188 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T15:38:45.210 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T15:38:45.214 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T15:38:45.255 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T15:38:45.261 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T15:38:45.262 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T15:38:45.281 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T15:38:45.284 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T15:38:45.318 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.329 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.331 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.331 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.341 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.343 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:38:45.344 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T15:38:45.352 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T15:38:45.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T15:38:45.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T15:38:45.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T15:38:45.415 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T15:38:45.423 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T15:38:45.424 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T15:38:45.451 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T15:38:45.454 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T15:38:45.473 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:45.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:45.622 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:45.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:45.719 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:38:45.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T15:38:45.794 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:45.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:45.899 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:45.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:46.030 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:38:46.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T15:38:46.099 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:46.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:38:46.210 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:46.257 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:38:46.300 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:38:46.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T15:38:46.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-24T15:38:46.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-04-24T15:38:46.373 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-24T15:38:46.419 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T15:38:46.425 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T15:38:46.454 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T15:38:46.734 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T15:38:46.743 INFO:teuthology.task.internal:Starting timer... 2024-04-24T15:38:46.743 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T15:38:46.753 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T15:38:46.761 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-24T15:38:46.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-24T15:38:46.824 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-24T15:38:46.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-04-24T15:38:46.899 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-04-24T15:38:46.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-24T15:38:46.968 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-24T15:38:46.970 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T15:38:46.970 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-24T15:38:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-24T15:38:47.021 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-04-24T15:38:47.048 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-04-24T15:38:47.049 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-24T15:38:47.069 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-24T15:38:47.069 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-24T15:38:47.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T15:38:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:47.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T15:38:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:47.129 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T15:38:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:38:47.156 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T15:38:47.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-24T15:38:47.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-04-24T15:38:47.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-24T15:38:47.271 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T15:38:47.282 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T15:38:47.287 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T15:38:47.410 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T15:38:47.425 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T15:38:47.427 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi086.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T15:49:34.759 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-24T15:49:34.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T15:49:34.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:49:34.849 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-24T15:49:34.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T15:49:34.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-24T15:49:34.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:49:35.000 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-24T15:49:35.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-24T15:49:35.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-24T15:49:35.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:49:35.148 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-24T15:49:35.179 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-24T15:49:35.180 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T15:49:35.190 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T15:49:35.190 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T15:49:35.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T15:49:35.193 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T15:49:35.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T15:49:35.195 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T15:49:35.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T15:49:35.230 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T15:49:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T15:49:35.244 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T15:49:35.248 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T15:49:35.257 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T15:49:35.275 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T15:49:35.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-04-24T15:49:35.328 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-04-24T15:49:35.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-24T15:49:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T15:49:35.347 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-24T15:49:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-24T15:49:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T15:49:35.364 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T15:49:35.371 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-24T15:49:35.381 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T15:49:35.388 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T15:49:35.404 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T15:49:35.429 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.433 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.470 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.474 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.475 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.479 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:49:35.480 INFO:teuthology.run_tasks:Running task install... 2024-04-24T15:49:35.488 DEBUG:teuthology.task.install:project ceph 2024-04-24T15:49:35.488 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T15:49:35.488 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'} 2024-04-24T15:49:35.489 INFO:teuthology.task.install:Using flavor: default 2024-04-24T15:49:35.494 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T15:49:35.495 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-04-24T15:49:35.495 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T15:49:35.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:49:35.497 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T15:49:35.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:49:35.498 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T15:49:35.498 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:49:35.642 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T15:49:35.642 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T15:49:35.661 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T15:49:35.661 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T15:49:35.718 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T15:49:35.719 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T15:49:35.954 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T15:49:35.954 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:49:35.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T15:49:35.957 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T15:49:35.957 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:49:35.957 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T15:49:35.990 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-04-24T15:49:35.991 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T15:49:35.992 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-04-24T15:49:35.992 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T15:49:36.012 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T15:49:36.012 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:49:36.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T15:49:36.047 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-04-24T15:49:36.047 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T15:49:36.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T15:49:36.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T15:49:36.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T15:49:36.143 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T15:49:36.151 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T15:49:36.183 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-04-24T15:49:36.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-04-24T15:49:36.211 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T15:49:36.213 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-04-24T15:49:36.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-04-24T15:49:36.243 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-24T15:49:36.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-24T15:49:36.579 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:36.594 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:36.682 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:38.016 INFO:teuthology.orchestra.run.smithi122.stdout:63 files removed 2024-04-24T15:49:38.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-04-24T15:49:38.106 INFO:teuthology.orchestra.run.smithi032.stdout:63 files removed 2024-04-24T15:49:38.120 INFO:teuthology.orchestra.run.smithi086.stdout:63 files removed 2024-04-24T15:49:38.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-04-24T15:49:38.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-04-24T15:49:38.402 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:38.506 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:38.546 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:49:40.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 94 kB/s | 78 kB 00:00 2024-04-24T15:49:40.873 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 91 kB/s | 78 kB 00:00 2024-04-24T15:49:40.908 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 93 kB/s | 78 kB 00:00 2024-04-24T15:49:41.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-04-24T15:49:41.690 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-04-24T15:49:41.700 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-04-24T15:49:42.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 2.3 kB/s | 1.4 kB 00:00 2024-04-24T15:49:42.397 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-04-24T15:49:42.399 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 2.1 kB/s | 1.4 kB 00:00 2024-04-24T15:49:42.442 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 2.0 kB/s | 1.4 kB 00:00 2024-04-24T15:49:42.558 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-24T15:49:42.601 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-24T15:49:43.317 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-04-24T15:49:44.301 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 16 MB 00:01 2024-04-24T15:49:44.444 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 16 MB 00:01 2024-04-24T15:49:46.932 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-04-24T15:49:47.534 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 17 MB/s | 8.8 MB 00:00 2024-04-24T15:49:47.859 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 440 kB/s | 24 kB 00:00 2024-04-24T15:49:48.469 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 416 kB/s | 24 kB 00:00 2024-04-24T15:49:49.195 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 60 MB 00:01 2024-04-24T15:49:49.962 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 60 MB 00:01 2024-04-24T15:49:50.235 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 MB/s | 60 MB 00:01 2024-04-24T15:50:00.611 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-04-24T15:50:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 67 MB 00:01 2024-04-24T15:50:02.662 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-04-24T15:50:11.187 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 15 MB/s | 8.8 MB 00:00 2024-04-24T15:50:14.480 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 11 MB/s | 8.8 MB 00:00 2024-04-24T15:50:16.868 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:50:16.870 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2024-04-24T15:50:16.870 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:50:16.870 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2024-04-24T15:50:16.870 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:50:16.870 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-24T15:50:16.871 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:50:16.872 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:Install 21 Packages 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 64 M 2024-04-24T15:50:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:50:17.586 INFO:teuthology.orchestra.run.smithi086.stdout:(1/23): ceph-base-17.2.7-915.g330067db.el8.x86_ 9.3 MB/s | 5.3 MB 00:00 2024-04-24T15:50:17.644 INFO:teuthology.orchestra.run.smithi086.stdout:(2/23): ceph-selinux-17.2.7-915.g330067db.el8.x 418 kB/s | 24 kB 00:00 2024-04-24T15:50:17.744 INFO:teuthology.orchestra.run.smithi086.stdout:(3/23): libcephfs2-17.2.7-915.g330067db.el8.x86 7.8 MB/s | 798 kB 00:00 2024-04-24T15:50:17.820 INFO:teuthology.orchestra.run.smithi086.stdout:(4/23): libradosstriper1-17.2.7-915.g330067db.e 6.6 MB/s | 505 kB 00:00 2024-04-24T15:50:18.262 INFO:teuthology.orchestra.run.smithi086.stdout:(5/23): librgw2-17.2.7-915.g330067db.el8.x86_64 13 MB/s | 5.6 MB 00:00 2024-04-24T15:50:18.329 INFO:teuthology.orchestra.run.smithi086.stdout:(6/23): python3-ceph-argparse-17.2.7-915.g33006 672 kB/s | 45 kB 00:00 2024-04-24T15:50:18.335 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout:=================================================================================================== 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout:=================================================================================================== 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T15:50:18.338 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T15:50:18.339 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout:=================================================================================================== 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout:Install 21 Packages 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:50:18.340 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 64 M 2024-04-24T15:50:18.341 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:50:18.387 INFO:teuthology.orchestra.run.smithi086.stdout:(7/23): python3-ceph-common-17.2.7-915.g330067d 2.0 MB/s | 117 kB 00:00 2024-04-24T15:50:18.571 INFO:teuthology.orchestra.run.smithi086.stdout:(8/23): ceph-radosgw-17.2.7-915.g330067db.el8.x 11 MB/s | 17 MB 00:01 2024-04-24T15:50:18.596 INFO:teuthology.orchestra.run.smithi086.stdout:(9/23): python3-cephfs-17.2.7-915.g330067db.el8 911 kB/s | 190 kB 00:00 2024-04-24T15:50:18.630 INFO:teuthology.orchestra.run.smithi086.stdout:(10/23): python3-rados-17.2.7-915.g330067db.el8 6.2 MB/s | 370 kB 00:00 2024-04-24T15:50:18.693 INFO:teuthology.orchestra.run.smithi086.stdout:(11/23): python3-rbd-17.2.7-915.g330067db.el8.x 3.6 MB/s | 358 kB 00:00 2024-04-24T15:50:18.714 INFO:teuthology.orchestra.run.smithi086.stdout:(12/23): python3-rgw-17.2.7-915.g330067db.el8.x 1.2 MB/s | 107 kB 00:00 2024-04-24T15:50:18.922 INFO:teuthology.orchestra.run.smithi086.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2024-04-24T15:50:18.989 INFO:teuthology.orchestra.run.smithi086.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2024-04-24T15:50:19.014 INFO:teuthology.orchestra.run.smithi086.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 817 kB/s | 75 kB 00:00 2024-04-24T15:50:19.105 INFO:teuthology.orchestra.run.smithi122.stdout:(1/23): ceph-base-17.2.7-915.g330067db.el8.x86_ 7.5 MB/s | 5.3 MB 00:00 2024-04-24T15:50:19.165 INFO:teuthology.orchestra.run.smithi086.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 9.9 MB/s | 1.7 MB 00:00 2024-04-24T15:50:19.172 INFO:teuthology.orchestra.run.smithi122.stdout:(2/23): ceph-selinux-17.2.7-915.g330067db.el8.x 365 kB/s | 24 kB 00:00 2024-04-24T15:50:19.190 INFO:teuthology.orchestra.run.smithi086.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 250 kB/s | 44 kB 00:00 2024-04-24T15:50:19.330 INFO:teuthology.orchestra.run.smithi122.stdout:(3/23): libcephfs2-17.2.7-915.g330067db.el8.x86 4.9 MB/s | 798 kB 00:00 2024-04-24T15:50:19.349 INFO:teuthology.orchestra.run.smithi086.stdout:(18/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2024-04-24T15:50:19.374 INFO:teuthology.orchestra.run.smithi086.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-04-24T15:50:19.422 INFO:teuthology.orchestra.run.smithi122.stdout:(4/23): libradosstriper1-17.2.7-915.g330067db.e 5.4 MB/s | 505 kB 00:00 2024-04-24T15:50:19.516 INFO:teuthology.orchestra.run.smithi086.stdout:(20/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 282 kB/s | 47 kB 00:00 2024-04-24T15:50:19.922 INFO:teuthology.orchestra.run.smithi122.stdout:(5/23): ceph-radosgw-17.2.7-915.g330067db.el8.x 11 MB/s | 17 MB 00:01 2024-04-24T15:50:20.015 INFO:teuthology.orchestra.run.smithi122.stdout:(6/23): python3-ceph-argparse-17.2.7-915.g33006 487 kB/s | 45 kB 00:00 2024-04-24T15:50:20.075 INFO:teuthology.orchestra.run.smithi086.stdout:(21/23): librbd1-17.2.7-915.g330067db.el8.x86_6 6.4 MB/s | 3.6 MB 00:00 2024-04-24T15:50:20.132 INFO:teuthology.orchestra.run.smithi122.stdout:(7/23): librgw2-17.2.7-915.g330067db.el8.x86_64 7.9 MB/s | 5.6 MB 00:00 2024-04-24T15:50:20.157 INFO:teuthology.orchestra.run.smithi122.stdout:(8/23): python3-ceph-common-17.2.7-915.g330067d 824 kB/s | 117 kB 00:00 2024-04-24T15:50:20.182 INFO:teuthology.orchestra.run.smithi122.stdout:(9/23): python3-cephfs-17.2.7-915.g330067db.el8 3.7 MB/s | 190 kB 00:00 2024-04-24T15:50:20.199 INFO:teuthology.orchestra.run.smithi122.stdout:(10/23): python3-rados-17.2.7-915.g330067db.el8 8.6 MB/s | 370 kB 00:00 2024-04-24T15:50:20.216 INFO:teuthology.orchestra.run.smithi086.stdout:(22/23): librados2-17.2.7-915.g330067db.el8.x86 4.1 MB/s | 3.5 MB 00:00 2024-04-24T15:50:20.233 INFO:teuthology.orchestra.run.smithi122.stdout:(11/23): python3-rbd-17.2.7-915.g330067db.el8.x 7.0 MB/s | 358 kB 00:00 2024-04-24T15:50:20.258 INFO:teuthology.orchestra.run.smithi122.stdout:(12/23): python3-rgw-17.2.7-915.g330067db.el8.x 1.8 MB/s | 107 kB 00:00 2024-04-24T15:50:20.383 INFO:teuthology.orchestra.run.smithi122.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2024-04-24T15:50:20.409 INFO:teuthology.orchestra.run.smithi122.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2024-04-24T15:50:20.451 INFO:teuthology.orchestra.run.smithi122.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-04-24T15:50:20.567 INFO:teuthology.orchestra.run.smithi122.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2024-04-24T15:50:20.618 INFO:teuthology.orchestra.run.smithi122.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 263 kB/s | 44 kB 00:00 2024-04-24T15:50:20.735 INFO:teuthology.orchestra.run.smithi122.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-04-24T15:50:20.752 INFO:teuthology.orchestra.run.smithi122.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2024-04-24T15:50:20.902 INFO:teuthology.orchestra.run.smithi122.stdout:(20/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 282 kB/s | 47 kB 00:00 2024-04-24T15:50:21.019 INFO:teuthology.orchestra.run.smithi122.stdout:(21/23): librados2-17.2.7-915.g330067db.el8.x86 13 MB/s | 3.5 MB 00:00 2024-04-24T15:50:21.211 INFO:teuthology.orchestra.run.smithi122.stdout:(22/23): librbd1-17.2.7-915.g330067db.el8.x86_6 12 MB/s | 3.6 MB 00:00 2024-04-24T15:50:22.873 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout:=================================================================================================== 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout:=================================================================================================== 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:50:22.876 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T15:50:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T15:50:22.878 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:=================================================================================================== 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:Install 21 Packages 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 64 M 2024-04-24T15:50:22.879 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:50:24.033 INFO:teuthology.orchestra.run.smithi086.stdout:(23/23): ceph-common-17.2.7-915.g330067db.el8.x 3.4 MB/s | 24 MB 00:07 2024-04-24T15:50:24.036 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:50:24.036 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.9 MB/s | 64 MB 00:07 2024-04-24T15:50:24.086 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:50:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:50:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:50:24.389 INFO:teuthology.orchestra.run.smithi032.stdout:(1/23): ceph-base-17.2.7-915.g330067db.el8.x86_ 4.0 MB/s | 5.3 MB 00:01 2024-04-24T15:50:24.456 INFO:teuthology.orchestra.run.smithi032.stdout:(2/23): ceph-selinux-17.2.7-915.g330067db.el8.x 366 kB/s | 24 kB 00:00 2024-04-24T15:50:24.475 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:50:24.484 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:50:24.614 INFO:teuthology.orchestra.run.smithi032.stdout:(3/23): libcephfs2-17.2.7-915.g330067db.el8.x86 4.9 MB/s | 798 kB 00:00 2024-04-24T15:50:24.661 INFO:teuthology.orchestra.run.smithi122.stdout:(23/23): ceph-common-17.2.7-915.g330067db.el8.x 3.8 MB/s | 24 MB 00:06 2024-04-24T15:50:24.664 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:50:24.664 INFO:teuthology.orchestra.run.smithi122.stdout:Total 10 MB/s | 64 MB 00:06 2024-04-24T15:50:24.711 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:50:24.748 INFO:teuthology.orchestra.run.smithi032.stdout:(4/23): libradosstriper1-17.2.7-915.g330067db.e 3.7 MB/s | 505 kB 00:00 2024-04-24T15:50:24.756 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:50:24.756 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:50:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:50:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:50:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:50:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:(5/23): ceph-radosgw-17.2.7-915.g330067db.el8.x 6.5 MB/s | 17 MB 00:02 2024-04-24T15:50:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:(6/23): librgw2-17.2.7-915.g330067db.el8.x86_64 5.5 MB/s | 5.6 MB 00:01 2024-04-24T15:50:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:(7/23): python3-ceph-argparse-17.2.7-915.g33006 384 kB/s | 45 kB 00:00 2024-04-24T15:50:25.804 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:25.815 INFO:teuthology.orchestra.run.smithi032.stdout:(8/23): python3-ceph-common-17.2.7-915.g330067d 2.0 MB/s | 117 kB 00:00 2024-04-24T15:50:25.841 INFO:teuthology.orchestra.run.smithi032.stdout:(9/23): python3-cephfs-17.2.7-915.g330067db.el8 3.2 MB/s | 190 kB 00:00 2024-04-24T15:50:26.158 INFO:teuthology.orchestra.run.smithi032.stdout:(10/23): ceph-common-17.2.7-915.g330067db.el8.x 7.6 MB/s | 24 MB 00:03 2024-04-24T15:50:26.183 INFO:teuthology.orchestra.run.smithi032.stdout:(11/23): python3-rados-17.2.7-915.g330067db.el8 1.0 MB/s | 370 kB 00:00 2024-04-24T15:50:26.188 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:26.208 INFO:teuthology.orchestra.run.smithi032.stdout:(12/23): python3-rbd-17.2.7-915.g330067db.el8.x 976 kB/s | 358 kB 00:00 2024-04-24T15:50:26.221 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:50:26.225 INFO:teuthology.orchestra.run.smithi032.stdout:(13/23): python3-rgw-17.2.7-915.g330067db.el8.x 1.6 MB/s | 107 kB 00:00 2024-04-24T15:50:26.379 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:26.484 INFO:teuthology.orchestra.run.smithi032.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 214 kB/s | 59 kB 00:00 2024-04-24T15:50:26.498 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:26.509 INFO:teuthology.orchestra.run.smithi032.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 265 kB/s | 75 kB 00:00 2024-04-24T15:50:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 797 kB/s | 306 kB 00:00 2024-04-24T15:50:26.668 INFO:teuthology.orchestra.run.smithi032.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 277 kB/s | 44 kB 00:00 2024-04-24T15:50:26.685 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:50:26.769 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:26.776 INFO:teuthology.orchestra.run.smithi032.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-04-24T15:50:26.818 INFO:teuthology.orchestra.run.smithi032.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.2 MB/s | 1.7 MB 00:00 2024-04-24T15:50:26.852 INFO:teuthology.orchestra.run.smithi032.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2024-04-24T15:50:26.869 INFO:teuthology.orchestra.run.smithi032.stdout:(21/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 512 kB/s | 47 kB 00:00 2024-04-24T15:50:26.920 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2024-04-24T15:50:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:27.102 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:27.170 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:27.285 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:50:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2024-04-24T15:50:27.519 INFO:teuthology.orchestra.run.smithi032.stdout:(22/23): librbd1-17.2.7-915.g330067db.el8.x86_6 5.4 MB/s | 3.6 MB 00:00 2024-04-24T15:50:27.557 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:27.653 INFO:teuthology.orchestra.run.smithi032.stdout:(23/23): librados2-17.2.7-915.g330067db.el8.x86 4.2 MB/s | 3.5 MB 00:00 2024-04-24T15:50:27.658 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:50:27.659 INFO:teuthology.orchestra.run.smithi032.stdout:Total 13 MB/s | 64 MB 00:04 2024-04-24T15:50:27.687 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:27.712 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:50:27.729 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:27.763 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:50:27.763 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:50:27.830 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:28.010 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/25 2024-04-24T15:50:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:28.148 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:50:28.157 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:50:28.162 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:28.396 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:28.684 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/25 2024-04-24T15:50:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2024-04-24T15:50:29.071 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:29.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:29.284 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:29.355 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:50:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 11/25 2024-04-24T15:50:29.511 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:29.615 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:50:29.958 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2024-04-24T15:50:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2024-04-24T15:50:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 14/25 2024-04-24T15:50:30.137 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:30.153 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:30.213 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:30.264 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:30.281 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:30.340 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:30.364 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 11/25 2024-04-24T15:50:30.426 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:50:30.490 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-24T15:50:30.556 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:50:30.685 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2024-04-24T15:50:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2024-04-24T15:50:30.800 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:50:30.911 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:30.975 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-24T15:50:31.032 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 14/25 2024-04-24T15:50:31.146 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 19/25 2024-04-24T15:50:31.195 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:31.268 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:31.418 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-24T15:50:31.544 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:31.617 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:31.636 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2024-04-24T15:50:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/25 2024-04-24T15:50:31.895 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-24T15:50:32.053 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:32.054 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 19/25 2024-04-24T15:50:32.292 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-24T15:50:33.488 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2024-04-24T15:50:33.677 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:33.773 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 11/25 2024-04-24T15:50:34.115 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:50:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:50:34.628 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 14/25 2024-04-24T15:50:34.769 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:34.829 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T15:50:35.020 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-24T15:50:35.224 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2024-04-24T15:50:35.487 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-24T15:50:35.661 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 19/25 2024-04-24T15:50:35.711 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:35.927 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:35.953 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:36.162 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:36.736 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:36.823 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:36.913 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:37.095 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:37.262 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:39.956 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:40.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:40.840 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T15:50:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:41.225 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T15:50:41.418 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:56.451 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:56.451 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-04-24T15:50:56.451 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-04-24T15:50:56.451 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-04-24T15:50:56.452 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /run 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2024-04-24T15:50:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:50:58.112 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:50:58.189 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:50:58.189 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:50:58.394 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:50:58.394 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:50:58.414 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:50:58.502 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:50:58.502 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:50:58.711 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:50:58.711 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 1/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 7/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 8/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 9/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:59.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 11/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-24T15:50:59.069 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 24/25 2024-04-24T15:50:59.409 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 1/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 7/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 8/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 9/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 11/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-24T15:50:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-24T15:50:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 24/25 2024-04-24T15:51:03.585 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:51:03.585 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-24T15:51:03.585 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-24T15:51:03.585 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /run 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-24T15:51:03.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:51:05.543 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T15:51:05.543 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:51:05.793 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T15:51:05.793 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:51:06.571 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:51:06.571 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 1/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 7/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 8/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 9/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 11/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-24T15:51:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-24T15:51:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 24/25 2024-04-24T15:51:08.991 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:51:08.991 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:51:09.540 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T15:51:09.541 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:09.542 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:51:09.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-04-24T15:51:10.096 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:10.436 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:51:10.437 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:51:11.019 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-24T15:51:11.020 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:11.021 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:51:11.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-04-24T15:51:11.557 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:11.964 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 kB/s | 4.5 kB 00:00 2024-04-24T15:51:12.552 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 kB/s | 4.1 kB 00:00 2024-04-24T15:51:13.157 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-24T15:51:13.833 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:51:14.863 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================== 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================== 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:51:14.864 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================== 2024-04-24T15:51:14.865 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-04-24T15:51:14.865 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:14.865 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 54 M 2024-04-24T15:51:14.865 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 236 M 2024-04-24T15:51:14.865 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:51:14.905 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.8 MB/s | 70 kB 00:00 2024-04-24T15:51:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T15:51:15.021 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:51:15.072 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): libcephsqlite-17.2.7-915.g330067db.el8.x 872 kB/s | 179 kB 00:00 2024-04-24T15:51:15.114 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-04-24T15:51:15.231 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-04-24T15:51:15.340 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 835 kB/s | 188 kB 00:00 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.647 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.648 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.649 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:15.649 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T15:51:15.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:15.649 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:51:15.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-04-24T15:51:16.146 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:51:16.147 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-24T15:51:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:51:16.147 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-24T15:51:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:16.149 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 54 M 2024-04-24T15:51:16.149 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 236 M 2024-04-24T15:51:16.149 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:51:16.184 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2024-04-24T15:51:16.264 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:16.346 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-04-24T15:51:16.495 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2024-04-24T15:51:16.621 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): libcephsqlite-17.2.7-915.g330067db.el8.x 381 kB/s | 179 kB 00:00 2024-04-24T15:51:16.638 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-24T15:51:17.565 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-17.2.7-915.g330067db.el8.x86_6 19 MB/s | 53 MB 00:02 2024-04-24T15:51:17.567 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:17.567 INFO:teuthology.orchestra.run.smithi086.stdout:Total 20 MB/s | 54 MB 00:02 2024-04-24T15:51:17.606 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:51:17.627 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:51:17.627 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:51:17.915 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:51:17.924 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:51:18.141 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-24T15:51:18.406 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:51:18.531 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:18.610 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:18.834 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-24T15:51:18.852 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:51:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-24T15:51:19.188 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-24T15:51:19.339 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:21.562 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout:=========================================================================================== 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout:=========================================================================================== 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T15:51:21.563 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout:=========================================================================================== 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-24T15:51:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:21.565 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 54 M 2024-04-24T15:51:21.565 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 236 M 2024-04-24T15:51:21.565 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:51:21.589 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 3.1 MB/s | 70 kB 00:00 2024-04-24T15:51:21.771 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-04-24T15:51:21.846 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): libcephsqlite-17.2.7-915.g330067db.el8.x 640 kB/s | 179 kB 00:00 2024-04-24T15:51:21.897 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.6 MB/s | 202 kB 00:00 2024-04-24T15:51:22.055 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 903 kB/s | 188 kB 00:00 2024-04-24T15:51:24.289 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-test-17.2.7-915.g330067db.el8.x86_6 19 MB/s | 53 MB 00:02 2024-04-24T15:51:24.291 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:24.291 INFO:teuthology.orchestra.run.smithi032.stdout:Total 20 MB/s | 54 MB 00:02 2024-04-24T15:51:24.332 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:51:24.335 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:24.356 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:51:24.356 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:51:24.570 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:24.676 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:51:24.686 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 1/6 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2/6 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-24T15:51:24.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-24T15:51:25.224 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:51:25.396 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:25.486 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-24T15:51:25.940 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-24T15:51:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-24T15:51:26.296 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:26.838 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-17.2.7-915.g330067db.el8.x86_6 4.9 MB/s | 53 MB 00:10 2024-04-24T15:51:26.840 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:26.840 INFO:teuthology.orchestra.run.smithi122.stdout:Total 5.0 MB/s | 54 MB 00:10 2024-04-24T15:51:26.878 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:51:26.898 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:51:26.899 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:51:27.186 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:51:27.195 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:51:27.673 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:51:27.830 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:27.907 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-24T15:51:28.150 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-24T15:51:28.322 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-24T15:51:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-24T15:51:28.655 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:32.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 1/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-24T15:51:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-24T15:51:33.647 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T15:51:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 1/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-24T15:51:34.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-24T15:51:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-24T15:51:38.441 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-7.el8.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:38.991 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:51:39.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-04-24T15:51:39.514 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:41.320 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-24T15:51:41.964 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-24T15:51:43.219 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-24T15:51:43.219 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-7.el8.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:43.825 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:51:44.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-04-24T15:51:44.266 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================== 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================== 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T15:51:44.269 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T15:51:44.270 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================== 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2024-04-24T15:51:44.271 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:51:44.272 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 29 M 2024-04-24T15:51:44.272 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 113 M 2024-04-24T15:51:44.272 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:51:44.392 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:44.591 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-17.2.7-915.g330067db.el8.x86_64.rp 35 kB/s | 6.4 kB 00:00 2024-04-24T15:51:44.858 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mgr-17.2.7-915.g330067db.el8.x86_6 3.5 MB/s | 1.6 MB 00:00 2024-04-24T15:51:44.967 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mds-17.2.7-915.g330067db.el8.x86_6 4.1 MB/s | 2.3 MB 00:00 2024-04-24T15:51:45.025 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-17.2.7-915.g33006 4.0 MB/s | 240 kB 00:00 2024-04-24T15:51:45.326 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 145 kB/s | 44 kB 00:00 2024-04-24T15:51:45.576 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 741 kB/s | 185 kB 00:00 2024-04-24T15:51:45.716 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): ceph-mon-17.2.7-915.g330067db.el8.x86_6 3.6 MB/s | 4.0 MB 00:01 2024-04-24T15:51:45.743 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2024-04-24T15:51:45.994 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): ceph-osd-17.2.7-915.g330067db.el8.x86_6 15 MB/s | 17 MB 00:01 2024-04-24T15:51:46.019 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 40 kB/s | 11 kB 00:00 2024-04-24T15:51:46.127 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 132 kB/s | 18 kB 00:00 2024-04-24T15:51:46.253 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-cherrypy-18.4.0-1.el8.noarch.r 716 kB/s | 384 kB 00:00 2024-04-24T15:51:46.278 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 390 kB/s | 59 kB 00:00 2024-04-24T15:51:46.345 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-logutils-0.3.5-11.el8.noarch.r 149 kB/s | 49 kB 00:00 2024-04-24T15:51:46.353 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-24T15:51:46.396 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-04-24T15:51:46.421 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2024-04-24T15:51:46.471 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 149 kB/s | 19 kB 00:00 2024-04-24T15:51:46.521 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2024-04-24T15:51:46.546 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-04-24T15:51:46.705 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 118 kB/s | 27 kB 00:00 2024-04-24T15:51:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-04-24T15:51:46.756 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2024-04-24T15:51:46.956 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-04-24T15:51:46.996 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-24T15:51:46.997 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:51:47.085 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-24T15:51:47.098 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-markupsafe-0.23-19.el8.x86_64. 274 kB/s | 39 kB 00:00 2024-04-24T15:51:47.131 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-zc-lockfile-2.0-2.el8.noarch.r 57 kB/s | 23 kB 00:00 2024-04-24T15:51:47.273 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-webtest-2.0.33-1.el8.noarch.rp 150 kB/s | 85 kB 00:00 2024-04-24T15:51:47.317 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 472 kB/s | 103 kB 00:00 2024-04-24T15:51:47.382 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-mako-1.0.6-14.el8.noarch.rpm 627 kB/s | 157 kB 00:00 2024-04-24T15:51:47.466 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-pycparser-2.14-14.el8.noarch.r 764 kB/s | 109 kB 00:00 2024-04-24T15:51:47.491 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-pytz-2017.2-11.el8.noarch.rpm 249 kB/s | 54 kB 00:00 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-7.el8.x86_64 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T15:51:47.538 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T15:51:47.539 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:47.539 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:51:47.575 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2024-04-24T15:51:47.641 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 3.1 MB/s | 559 kB 00:00 2024-04-24T15:51:47.643 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:47.643 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.6 MB/s | 29 MB 00:03 2024-04-24T15:51:47.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-04-24T15:51:47.734 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:51:47.770 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:51:47.771 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:51:47.989 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:51:47.997 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:51:48.039 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:51:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:51:49.390 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-24T15:51:49.613 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-24T15:51:49.794 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================================== 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================================== 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T15:51:49.798 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T15:51:49.799 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:49.800 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================================== 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout:Install 32 Packages 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 29 M 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 113 M 2024-04-24T15:51:49.801 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:51:49.845 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-24T15:51:49.935 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-24T15:51:50.075 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-17.2.7-915.g330067db.el8.x86_64.rp 49 kB/s | 6.4 kB 00:00 2024-04-24T15:51:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-24T15:51:50.225 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-mgr-17.2.7-915.g330067db.el8.x86_6 5.6 MB/s | 1.6 MB 00:00 2024-04-24T15:51:50.404 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-24T15:51:50.492 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): ceph-mon-17.2.7-915.g330067db.el8.x86_6 9.7 MB/s | 4.0 MB 00:00 2024-04-24T15:51:50.561 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): ceph-mgr-modules-core-17.2.7-915.g33006 3.5 MB/s | 240 kB 00:00 2024-04-24T15:51:50.593 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:51:50.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-24T15:51:50.777 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-24T15:51:51.001 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 99 kB/s | 44 kB 00:00 2024-04-24T15:51:51.015 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-24T15:51:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): ceph-osd-17.2.7-915.g330067db.el8.x86_6 18 MB/s | 17 MB 00:00 2024-04-24T15:51:51.180 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-24T15:51:51.301 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2024-04-24T15:51:51.327 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 569 kB/s | 185 kB 00:00 2024-04-24T15:51:51.393 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): ceph-mds-17.2.7-915.g330067db.el8.x86_6 1.6 MB/s | 2.3 MB 00:01 2024-04-24T15:51:51.419 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-04-24T15:51:51.429 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-24T15:51:51.452 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-04-24T15:51:51.519 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 880 kB/s | 59 kB 00:00 2024-04-24T15:51:51.544 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 389 kB/s | 49 kB 00:00 2024-04-24T15:51:51.586 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): python3-jaraco-functools-2.0-4.el8.noa 91 kB/s | 18 kB 00:00 2024-04-24T15:51:51.612 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-04-24T15:51:51.637 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-04-24T15:51:51.662 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 478 kB/s | 24 kB 00:00 2024-04-24T15:51:51.688 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 582 kB/s | 29 kB 00:00 2024-04-24T15:51:51.713 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 551 kB/s | 27 kB 00:00 2024-04-24T15:51:51.747 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-24T15:51:51.796 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2024-04-24T15:51:51.847 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-04-24T15:51:51.881 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): python3-simplegeneric-0.8.1-17.el8.noa 64 kB/s | 19 kB 00:00 2024-04-24T15:51:51.906 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 393 kB/s | 23 kB 00:00 2024-04-24T15:51:51.968 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-24T15:51:51.990 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): python3-waitress-1.4.3-1.el8.noarch.rp 847 kB/s | 255 kB 00:00 2024-04-24T15:51:52.040 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): python3-markupsafe-0.23-19.el8.x86_64. 291 kB/s | 39 kB 00:00 2024-04-24T15:51:52.115 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.9 MB/s | 457 kB 00:00 2024-04-24T15:51:52.140 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-04-24T15:51:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 589 kB/s | 103 kB 00:00 2024-04-24T15:51:52.232 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): python3-pytz-2017.2-11.el8.noarch.rpm 464 kB/s | 54 kB 00:00 2024-04-24T15:51:52.249 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-04-24T15:51:52.303 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-24T15:51:52.308 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2024-04-24T15:51:52.391 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 3.4 MB/s | 559 kB 00:00 2024-04-24T15:51:52.393 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:52.393 INFO:teuthology.orchestra.run.smithi032.stdout:Total 11 MB/s | 29 MB 00:02 2024-04-24T15:51:52.496 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:51:52.537 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:51:52.538 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:51:52.784 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:51:52.793 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:51:52.929 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================================== 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================================== 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T15:51:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-24T15:51:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T15:51:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================================== 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout:Install 32 Packages 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:51:52.935 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 29 M 2024-04-24T15:51:52.936 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 113 M 2024-04-24T15:51:52.936 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:51:53.237 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-17.2.7-915.g330067db.el8.x86_64.rp 36 kB/s | 6.4 kB 00:00 2024-04-24T15:51:53.371 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-mds-17.2.7-915.g330067db.el8.x86_6 7.4 MB/s | 2.3 MB 00:00 2024-04-24T15:51:53.721 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): ceph-mgr-17.2.7-915.g330067db.el8.x86_6 2.4 MB/s | 1.6 MB 00:00 2024-04-24T15:51:53.838 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): ceph-mgr-modules-core-17.2.7-915.g33006 2.0 MB/s | 240 kB 00:00 2024-04-24T15:51:53.964 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 349 kB/s | 44 kB 00:00 2024-04-24T15:51:53.989 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:51:54.047 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.2 MB/s | 185 kB 00:00 2024-04-24T15:51:54.206 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): ceph-osd-17.2.7-915.g330067db.el8.x86_6 21 MB/s | 17 MB 00:00 2024-04-24T15:51:54.223 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 984 kB/s | 173 kB 00:00 2024-04-24T15:51:54.249 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-24T15:51:54.256 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 332 kB/s | 11 kB 00:00 2024-04-24T15:51:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 538 kB/s | 18 kB 00:00 2024-04-24T15:51:54.323 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2024-04-24T15:51:54.357 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2024-04-24T15:51:54.363 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-24T15:51:54.408 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-04-24T15:51:54.433 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-04-24T15:51:54.458 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2024-04-24T15:51:54.484 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2024-04-24T15:51:54.509 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2024-04-24T15:51:54.520 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-24T15:51:54.534 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 580 kB/s | 29 kB 00:00 2024-04-24T15:51:54.559 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 549 kB/s | 27 kB 00:00 2024-04-24T15:51:54.585 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2024-04-24T15:51:54.610 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.9 MB/s | 251 kB 00:00 2024-04-24T15:51:54.635 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-04-24T15:51:54.661 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2024-04-24T15:51:54.714 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-24T15:51:54.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-24T15:51:54.836 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-04-24T15:51:54.861 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): python3-markupsafe-0.23-19.el8.x86_64. 194 kB/s | 39 kB 00:00 2024-04-24T15:51:54.912 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-24T15:51:54.945 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 955 kB/s | 103 kB 00:00 2024-04-24T15:51:55.004 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-04-24T15:51:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-24T15:51:55.112 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): ceph-mon-17.2.7-915.g330067db.el8.x86_ 2.1 MB/s | 4.0 MB 00:01 2024-04-24T15:51:55.138 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): python3-pytz-2017.2-11.el8.noarch.rpm 282 kB/s | 54 kB 00:00 2024-04-24T15:51:55.155 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-24T15:51:55.179 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 621 kB/s | 109 kB 00:00 2024-04-24T15:51:55.221 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2024-04-24T15:51:55.263 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): python3-cryptography-3.2.1-7.el8_9.x86 4.6 MB/s | 559 kB 00:00 2024-04-24T15:51:55.267 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:51:55.267 INFO:teuthology.orchestra.run.smithi122.stdout:Total 12 MB/s | 29 MB 00:02 2024-04-24T15:51:55.279 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-24T15:51:55.361 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:51:55.398 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:51:55.398 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:51:55.433 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-24T15:51:55.509 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-24T15:51:55.615 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:51:55.624 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:51:55.644 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-24T15:51:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-24T15:51:55.812 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-24T15:51:55.915 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-24T15:51:56.138 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-24T15:51:56.176 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-24T15:51:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-24T15:51:56.415 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-24T15:51:56.636 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-24T15:51:56.682 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-24T15:51:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:51:56.940 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-24T15:51:56.942 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-24T15:51:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-24T15:51:57.172 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-24T15:51:57.286 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-24T15:51:57.335 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-24T15:51:57.378 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-24T15:51:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-24T15:51:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 27/32 2024-04-24T15:51:57.750 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-24T15:51:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-24T15:51:57.887 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:51:58.012 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-24T15:51:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-24T15:51:58.184 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-24T15:51:58.306 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-24T15:51:58.335 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-24T15:51:58.495 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-24T15:51:58.548 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-24T15:51:58.679 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-24T15:51:58.696 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-24T15:51:58.885 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-24T15:51:58.962 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-24T15:51:59.298 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-24T15:51:59.403 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:51:59.517 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-24T15:51:59.645 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:51:59.734 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-24T15:51:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-24T15:51:59.949 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-24T15:52:00.047 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:52:00.113 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-24T15:52:00.209 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:00.310 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-24T15:52:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:00.495 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-24T15:52:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:00.658 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-24T15:52:00.758 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-24T15:52:01.087 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-24T15:52:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/32 2024-04-24T15:52:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 3/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 4/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 5/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 6/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-24T15:52:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-24T15:52:01.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-24T15:52:01.134 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-24T15:52:01.134 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-24T15:52:01.134 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-24T15:52:01.134 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-24T15:52:01.135 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-24T15:52:01.135 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-24T15:52:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-24T15:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-24T15:52:01.755 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-24T15:52:01.790 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-24T15:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-24T15:52:02.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-24T15:52:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-24T15:52:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-24T15:52:02.622 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-24T15:52:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-24T15:52:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-24T15:52:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-24T15:52:03.412 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 27/32 2024-04-24T15:52:03.829 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 27/32 2024-04-24T15:52:04.049 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:52:05.382 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:52:05.734 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:52:05.957 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:52:06.406 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:52:06.566 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:06.808 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:06.903 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T15:52:07.016 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:07.041 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:07.089 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:52:07.158 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:07.490 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T15:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 3/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 4/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 5/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 6/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-24T15:52:07.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-24T15:52:07.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-24T15:52:07.664 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-24T15:52:07.664 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-24T15:52:07.664 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-24T15:52:07.664 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-24T15:52:07.664 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-24T15:52:07.867 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T15:52:08.034 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:08.052 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T15:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T15:52:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 3/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 4/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 5/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 6/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-24T15:52:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-24T15:52:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-24T15:52:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-24T15:52:12.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-24T15:52:12.925 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:52:13.484 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:13.484 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T15:52:13.485 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T15:52:13.486 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:13.487 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:52:13.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-04-24T15:52:14.035 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:15.895 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-24T15:52:16.527 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:52:18.729 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:52:18.914 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:52:18.916 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:52:18.916 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:52:19.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-04-24T15:52:19.123 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-24T15:52:19.123 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:52:19.279 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-24T15:52:19.279 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:52:19.402 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T15:52:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T15:52:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:19.670 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:52:19.866 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-04-24T15:52:19.908 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T15:52:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T15:52:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:19.911 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:52:20.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-04-24T15:52:20.215 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:20.506 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:21.310 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-24T15:52:21.877 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-24T15:52:21.922 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-24T15:52:22.436 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-24T15:52:22.574 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:52:23.109 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 kB/s | 4.1 kB 00:00 2024-04-24T15:52:24.299 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:52:24.299 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:24.299 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:52:24.299 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:52:24.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:24.301 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-04-24T15:52:24.301 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 372 k 2024-04-24T15:52:24.301 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:52:24.474 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.7-915.g330067db.el8.noarch.rpm 472 kB/s | 82 kB 00:00 2024-04-24T15:52:24.475 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:24.475 INFO:teuthology.orchestra.run.smithi086.stdout:Total 469 kB/s | 82 kB 00:00 2024-04-24T15:52:24.475 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:52:24.483 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:52:24.483 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:52:24.485 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:52:24.494 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:52:24.536 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:52:24.750 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:52:24.933 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:52:24.934 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:52:24.934 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:52:25.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-04-24T15:52:25.438 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:25.613 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:52:25.820 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:52:25.821 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:52:25.821 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:52:25.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-04-24T15:52:26.395 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:26.487 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:26.654 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:27.321 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-24T15:52:27.946 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-04-24T15:52:28.326 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 23 kB/s | 4.5 kB 00:00 2024-04-24T15:52:29.060 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-24T15:52:30.280 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:52:30.280 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:30.280 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:30.281 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-04-24T15:52:30.282 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 372 k 2024-04-24T15:52:30.282 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:52:30.553 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-17.2.7-915.g330067db.el8.noarch.rpm 302 kB/s | 82 kB 00:00 2024-04-24T15:52:30.553 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:30.554 INFO:teuthology.orchestra.run.smithi122.stdout:Total 300 kB/s | 82 kB 00:00 2024-04-24T15:52:30.554 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:52:30.562 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:52:30.562 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:52:30.564 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:52:30.573 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:52:30.615 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:52:31.759 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:52:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:31.761 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 82 k 2024-04-24T15:52:31.761 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 372 k 2024-04-24T15:52:31.761 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:52:32.039 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-17.2.7-915.g330067db.el8.noarch.rpm 295 kB/s | 82 kB 00:00 2024-04-24T15:52:32.040 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:32.040 INFO:teuthology.orchestra.run.smithi032.stdout:Total 293 kB/s | 82 kB 00:00 2024-04-24T15:52:32.041 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:52:32.051 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:52:32.052 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:52:32.054 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:52:32.064 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:52:32.116 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:52:32.499 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:32.657 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:32.858 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:34.509 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:34.725 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:34.977 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:38.460 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:38.460 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:52:39.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:39.003 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:52:39.003 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:39.003 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:39.003 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:52:39.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T15:52:39.533 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:41.345 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-24T15:52:41.914 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-24T15:52:43.227 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:43.227 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:52:43.765 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:52:43.766 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:43.766 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:52:43.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T15:52:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:52:44.148 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:52:44.230 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:52:44.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:44.232 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 161 k 2024-04-24T15:52:44.232 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-04-24T15:52:44.232 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:52:44.293 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:44.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.7-915.g330067d 774 kB/s | 161 kB 00:00 2024-04-24T15:52:44.441 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:44.441 INFO:teuthology.orchestra.run.smithi086.stdout:Total 769 kB/s | 161 kB 00:00 2024-04-24T15:52:44.442 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:52:44.452 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:52:44.452 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:52:44.506 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:52:44.514 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:52:44.629 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:52:44.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:44.745 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:52:44.745 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:52:44.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:44.745 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:52:44.762 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:44.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T15:52:45.153 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:45.334 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:46.168 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-24T15:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-04-24T15:52:47.308 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-24T15:52:48.011 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-04-24T15:52:49.100 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:52:49.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:52:49.102 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 161 k 2024-04-24T15:52:49.102 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 494 k 2024-04-24T15:52:49.102 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:52:49.421 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-17.2.7-915.g330067d 506 kB/s | 161 kB 00:00 2024-04-24T15:52:49.422 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:49.422 INFO:teuthology.orchestra.run.smithi122.stdout:Total 504 kB/s | 161 kB 00:00 2024-04-24T15:52:49.422 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:52:49.433 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:52:49.433 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:52:49.487 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:52:49.495 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:52:49.609 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:52:49.745 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:50.126 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:50.723 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:52:50.724 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:52:50.725 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:52:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 161 k 2024-04-24T15:52:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 494 k 2024-04-24T15:52:50.725 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:52:50.938 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-17.2.7-915.g330067d 761 kB/s | 161 kB 00:00 2024-04-24T15:52:50.938 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:52:50.939 INFO:teuthology.orchestra.run.smithi032.stdout:Total 755 kB/s | 161 kB 00:00 2024-04-24T15:52:50.939 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:52:50.952 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:52:50.953 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:52:51.017 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:52:51.026 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:52:51.151 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:52:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:51.764 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:54.296 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:52:54.296 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:52:54.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:54.832 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:52:54.832 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:52:54.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:52:54.832 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:52:54.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-04-24T15:52:55.337 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:52:57.519 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-24T15:52:58.118 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-24T15:53:00.250 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:53:00.437 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:53:00.438 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:53:00.438 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:53:00.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T15:53:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:53:00.897 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:53:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:01.442 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:53:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:53:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:01.442 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:53:01.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-04-24T15:53:01.957 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-24T15:53:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T15:53:03.269 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:53:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:53:03.889 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:03.890 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:53:03.890 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:53:03.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:03.890 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:53:03.940 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-24T15:53:04.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-04-24T15:53:04.470 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:04.605 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-24T15:53:05.722 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================== 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================== 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T15:53:05.723 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================== 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-04-24T15:53:05.724 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:05.725 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2024-04-24T15:53:05.725 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 80 M 2024-04-24T15:53:05.725 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:53:06.039 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-grafana-dashboards-17.2.7-915.g3300 175 kB/s | 24 kB 00:00 2024-04-24T15:53:06.197 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-prometheus-alerts-17.2.7-915.g33006 49 kB/s | 14 kB 00:00 2024-04-24T15:53:06.318 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 kB/s | 4.5 kB 00:00 2024-04-24T15:53:06.364 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-mgr-dashboard-17.2.7-915.g330067db. 7.2 MB/s | 3.4 MB 00:00 2024-04-24T15:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 301 kB/s | 43 kB 00:00 2024-04-24T15:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 68 kB/s | 34 kB 00:00 2024-04-24T15:53:06.747 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:53:06.924 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 271 kB/s | 196 kB 00:00 2024-04-24T15:53:06.927 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:06.927 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.0 MB/s | 3.7 MB 00:01 2024-04-24T15:53:06.934 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:53:06.935 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:53:06.936 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:53:06.941 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:53:06.951 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:53:06.952 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:53:07.030 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:53:07.039 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:53:07.041 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:53:07.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T15:53:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:53:07.435 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:07.510 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2024-04-24T15:53:07.714 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2024-04-24T15:53:07.928 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2024-04-24T15:53:08.074 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 4/6 2024-04-24T15:53:09.153 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-24T15:53:09.209 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 5/6 2024-04-24T15:53:09.478 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:09.546 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:53:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 1/6 2024-04-24T15:53:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2/6 2024-04-24T15:53:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 3/6 2024-04-24T15:53:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2024-04-24T15:53:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2024-04-24T15:53:09.759 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:53:09.760 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:53:09.760 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:53:09.791 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-24T15:53:09.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T15:53:10.302 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-24T15:53:12.235 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:53:12.236 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================================== 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================================== 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:12.237 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================================== 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.7 M 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 80 M 2024-04-24T15:53:12.238 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:53:12.516 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): ceph-grafana-dashboards-17.2.7-915.g3300 109 kB/s | 24 kB 00:00 2024-04-24T15:53:12.541 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): ceph-prometheus-alerts-17.2.7-915.g33006 58 kB/s | 14 kB 00:00 2024-04-24T15:53:12.667 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 224 kB/s | 34 kB 00:00 2024-04-24T15:53:12.717 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-04-24T15:53:12.834 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2024-04-24T15:53:12.998 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-24T15:53:13.193 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-mgr-dashboard-17.2.7-915.g330067db. 3.7 MB/s | 3.4 MB 00:00 2024-04-24T15:53:13.196 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:13.196 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.8 MB/s | 3.7 MB 00:00 2024-04-24T15:53:13.207 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:53:13.217 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:53:13.217 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:53:13.289 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:53:13.297 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:53:13.567 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:53:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2024-04-24T15:53:13.998 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2024-04-24T15:53:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2024-04-24T15:53:14.335 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 4/6 2024-04-24T15:53:15.442 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 5/6 2024-04-24T15:53:15.672 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:15.809 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:53:15.810 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================================== 2024-04-24T15:53:15.810 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:53:15.810 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================================== 2024-04-24T15:53:15.810 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:53:15.810 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================================== 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-24T15:53:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:15.812 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.7 M 2024-04-24T15:53:15.812 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 80 M 2024-04-24T15:53:15.812 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:53:15.922 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:15.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 1/6 2024-04-24T15:53:15.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2/6 2024-04-24T15:53:15.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 3/6 2024-04-24T15:53:15.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2024-04-24T15:53:15.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2024-04-24T15:53:16.025 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): ceph-grafana-dashboards-17.2.7-915.g3300 157 kB/s | 24 kB 00:00 2024-04-24T15:53:16.117 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): ceph-prometheus-alerts-17.2.7-915.g33006 59 kB/s | 14 kB 00:00 2024-04-24T15:53:16.310 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 118 kB/s | 34 kB 00:00 2024-04-24T15:53:16.434 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): ceph-mgr-dashboard-17.2.7-915.g330067db. 6.0 MB/s | 3.4 MB 00:00 2024-04-24T15:53:16.459 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 574 kB/s | 196 kB 00:00 2024-04-24T15:53:16.632 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 133 kB/s | 43 kB 00:00 2024-04-24T15:53:16.636 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:16.636 INFO:teuthology.orchestra.run.smithi032.stdout:Total 4.4 MB/s | 3.7 MB 00:00 2024-04-24T15:53:16.651 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:53:16.662 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:53:16.662 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:53:16.749 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:53:16.760 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:53:17.067 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:53:17.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2024-04-24T15:53:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2024-04-24T15:53:17.685 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2024-04-24T15:53:17.862 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 4/6 2024-04-24T15:53:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 5/6 2024-04-24T15:53:19.330 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T15:53:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 1/6 2024-04-24T15:53:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2/6 2024-04-24T15:53:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 3/6 2024-04-24T15:53:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2024-04-24T15:53:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2024-04-24T15:53:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2024-04-24T15:53:20.247 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:20.802 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:53:20.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T15:53:21.330 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:23.278 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-24T15:53:23.897 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-24T15:53:26.192 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:53:26.193 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2024-04-24T15:53:26.193 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:53:26.193 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2024-04-24T15:53:26.193 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2024-04-24T15:53:26.194 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:26.195 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 36 M 2024-04-24T15:53:26.195 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 207 M 2024-04-24T15:53:26.195 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:53:26.380 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2024-04-24T15:53:26.414 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.0 MB/s | 225 kB 00:00 2024-04-24T15:53:26.581 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 26 MB/s | 5.2 MB 00:00 2024-04-24T15:53:26.723 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-04-24T15:53:26.881 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-04-24T15:53:27.065 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): python3-scipy-1.0.0-21.module+el8.9.0+20 28 MB/s | 14 MB 00:00 2024-04-24T15:53:27.115 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 738 kB/s | 172 kB 00:00 2024-04-24T15:53:27.232 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.7-915 7.2 MB/s | 7.4 MB 00:01 2024-04-24T15:53:27.233 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:27.233 INFO:teuthology.orchestra.run.smithi086.stdout:Total 35 MB/s | 36 MB 00:01 2024-04-24T15:53:27.479 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:53:27.497 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:53:27.498 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:53:27.806 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:53:27.814 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:53:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:53:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2024-04-24T15:53:28.548 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:53:28.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:28.708 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:28.856 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:29.114 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:29.115 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:53:29.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T15:53:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:29.622 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:29.630 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:30.212 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:30.422 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:31.107 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:31.219 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-24T15:53:31.409 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T15:53:31.639 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:31.840 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-04-24T15:53:33.108 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2024-04-24T15:53:33.108 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:53:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:33.743 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:53:33.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T15:53:34.150 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:53:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================================= 2024-04-24T15:53:34.151 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:53:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================================= 2024-04-24T15:53:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:53:34.151 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================================= 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout:Install 8 Packages 2024-04-24T15:53:34.152 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:34.153 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 36 M 2024-04-24T15:53:34.153 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 207 M 2024-04-24T15:53:34.153 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:53:34.317 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:34.367 INFO:teuthology.orchestra.run.smithi122.stdout:(1/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.0 MB/s | 225 kB 00:00 2024-04-24T15:53:34.450 INFO:teuthology.orchestra.run.smithi122.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2024-04-24T15:53:34.651 INFO:teuthology.orchestra.run.smithi122.stdout:(3/8): openblas-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2024-04-24T15:53:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-24T15:53:34.784 INFO:teuthology.orchestra.run.smithi122.stdout:(4/8): ceph-mgr-diskprediction-local-17.2.7-915 12 MB/s | 7.4 MB 00:00 2024-04-24T15:53:34.838 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:34.859 INFO:teuthology.orchestra.run.smithi122.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 11 MB/s | 5.2 MB 00:00 2024-04-24T15:53:34.985 INFO:teuthology.orchestra.run.smithi122.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-04-24T15:53:35.019 INFO:teuthology.orchestra.run.smithi122.stdout:(7/8): libgfortran-8.5.0-20.el8.x86_64.rpm 2.7 MB/s | 645 kB 00:00 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 1/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-24T15:53:35.129 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-24T15:53:35.202 INFO:teuthology.orchestra.run.smithi122.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 25 MB/s | 14 MB 00:00 2024-04-24T15:53:35.203 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:35.203 INFO:teuthology.orchestra.run.smithi122.stdout:Total 34 MB/s | 36 MB 00:01 2024-04-24T15:53:35.448 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:53:35.467 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:53:35.467 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:53:35.777 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:53:35.786 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:53:36.195 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 4.5 kB 00:00 2024-04-24T15:53:36.347 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:53:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:36.656 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:36.797 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:36.903 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-24T15:53:37.437 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:37.597 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:38.182 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:38.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:39.063 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:39.375 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T15:53:39.564 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:39.617 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================================= 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================================= 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T15:53:39.618 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================================= 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout:Install 8 Packages 2024-04-24T15:53:39.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:53:39.620 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 36 M 2024-04-24T15:53:39.620 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 207 M 2024-04-24T15:53:39.620 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:53:39.804 INFO:teuthology.orchestra.run.smithi032.stdout:(1/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.2 MB/s | 225 kB 00:00 2024-04-24T15:53:39.917 INFO:teuthology.orchestra.run.smithi032.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2024-04-24T15:53:40.042 INFO:teuthology.orchestra.run.smithi032.stdout:(3/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.2 MB 00:00 2024-04-24T15:53:40.176 INFO:teuthology.orchestra.run.smithi032.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-04-24T15:53:40.268 INFO:teuthology.orchestra.run.smithi032.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 6.9 MB/s | 645 kB 00:00 2024-04-24T15:53:40.418 INFO:teuthology.orchestra.run.smithi032.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-04-24T15:53:40.560 INFO:teuthology.orchestra.run.smithi032.stdout:(7/8): ceph-mgr-diskprediction-local-17.2.7-915 7.9 MB/s | 7.4 MB 00:00 2024-04-24T15:53:40.760 INFO:teuthology.orchestra.run.smithi032.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 19 MB/s | 14 MB 00:00 2024-04-24T15:53:40.762 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:40.762 INFO:teuthology.orchestra.run.smithi032.stdout:Total 32 MB/s | 36 MB 00:01 2024-04-24T15:53:41.033 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:53:41.053 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:53:41.053 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:53:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:41.398 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:53:41.408 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:53:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:53:42.195 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:42.346 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-24T15:53:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:42.561 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-24T15:53:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:43.021 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:43.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 1/8 2024-04-24T15:53:43.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-24T15:53:43.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-24T15:53:43.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:43.022 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-24T15:53:43.022 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-24T15:53:43.022 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-24T15:53:43.222 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-24T15:53:43.445 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-24T15:53:44.511 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:45.271 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:45.752 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T15:53:46.003 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T15:53:48.988 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-24T15:53:48.988 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-24T15:53:49.578 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:49.579 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:53:49.606 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-24T15:53:49.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-04-24T15:53:49.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:50.096 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T15:53:50.096 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 1/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-24T15:53:50.097 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-24T15:53:50.110 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:51.892 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-24T15:53:52.496 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:53:54.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-24T15:53:54.255 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:53:54.803 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:53:54.805 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2024-04-24T15:53:54.805 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:53:54.805 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2024-04-24T15:53:54.805 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-24T15:53:54.806 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-04-24T15:53:54.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:53:54.808 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2024-04-24T15:53:54.808 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 25 M 2024-04-24T15:53:54.808 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-24T15:53:54.814 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:53:54.815 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:53:54.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-04-24T15:53:55.314 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 109 kB/s | 33 kB 00:00 2024-04-24T15:53:55.338 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 50 kB/s | 16 kB 00:00 2024-04-24T15:53:55.347 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:53:55.397 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2024-04-24T15:53:55.439 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2024-04-24T15:53:55.506 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): ceph-mgr-rook-17.2.7-915.g330067db.el8. 96 kB/s | 47 kB 00:00 2024-04-24T15:53:55.532 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 665 kB/s | 61 kB 00:00 2024-04-24T15:53:55.640 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-jsonpointer-1.10-11.el8.noarch. 150 kB/s | 20 kB 00:00 2024-04-24T15:53:55.674 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.5 MB/s | 1.2 MB 00:00 2024-04-24T15:53:55.699 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 659 kB/s | 110 kB 00:00 2024-04-24T15:53:55.758 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 227 kB/s | 27 kB 00:00 2024-04-24T15:53:55.885 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 862 kB/s | 155 kB 00:00 2024-04-24T15:53:55.916 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 269 kB/s | 43 kB 00:00 2024-04-24T15:53:55.933 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 488 kB/s | 126 kB 00:00 2024-04-24T15:53:55.934 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:53:55.934 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-04-24T15:53:56.001 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:53:56.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:53:56.016 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:53:56.097 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:53:56.106 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:53:56.609 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:53:56.877 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-24T15:53:57.003 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-24T15:53:57.137 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2024-04-24T15:53:57.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2024-04-24T15:53:57.614 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2024-04-24T15:53:57.650 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-24T15:53:57.824 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2024-04-24T15:53:58.025 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2024-04-24T15:53:58.205 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2024-04-24T15:53:58.451 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2024-04-24T15:53:58.642 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-24T15:53:58.925 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2024-04-24T15:53:59.993 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2024-04-24T15:54:00.083 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:54:00.084 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================= 2024-04-24T15:54:00.084 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================= 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-24T15:54:00.085 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================================= 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-04-24T15:54:00.086 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:00.087 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.9 M 2024-04-24T15:54:00.087 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 25 M 2024-04-24T15:54:00.087 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:54:00.401 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 115 kB/s | 16 kB 00:00 2024-04-24T15:54:00.419 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 207 kB/s | 33 kB 00:00 2024-04-24T15:54:00.510 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 741 kB/s | 81 kB 00:00 2024-04-24T15:54:00.528 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): ceph-mgr-rook-17.2.7-915.g330067db.el8. 176 kB/s | 47 kB 00:00 2024-04-24T15:54:00.570 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-04-24T15:54:00.687 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 4.4 MB/s | 1.2 MB 00:00 2024-04-24T15:54:00.703 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-websocket-client-0.56.0-5.el8.n 349 kB/s | 61 kB 00:00 2024-04-24T15:54:00.729 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-jsonpointer-1.10-11.el8.noarch. 126 kB/s | 20 kB 00:00 2024-04-24T15:54:00.771 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2024-04-24T15:54:00.871 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 164 kB/s | 27 kB 00:00 2024-04-24T15:54:00.905 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 720 kB/s | 126 kB 00:00 2024-04-24T15:54:00.947 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 883 kB/s | 155 kB 00:00 2024-04-24T15:54:01.012 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2024-04-24T15:54:01.055 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 232 kB/s | 43 kB 00:00 2024-04-24T15:54:01.056 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:01.056 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-04-24T15:54:01.119 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:54:01.134 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:54:01.135 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:54:01.215 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:54:01.224 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:54:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 1/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2024-04-24T15:54:01.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2024-04-24T15:54:01.614 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2024-04-24T15:54:01.614 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2024-04-24T15:54:01.614 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2024-04-24T15:54:01.614 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2024-04-24T15:54:01.728 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:54:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-24T15:54:02.171 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2024-04-24T15:54:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2024-04-24T15:54:02.689 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2024-04-24T15:54:02.916 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2024-04-24T15:54:03.092 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2024-04-24T15:54:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2024-04-24T15:54:03.434 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2024-04-24T15:54:03.650 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-24T15:54:03.669 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-24T15:54:03.669 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:54:03.907 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2024-04-24T15:54:04.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:04.331 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:54:04.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-04-24T15:54:04.948 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2024-04-24T15:54:05.099 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:05.194 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2024-04-24T15:54:05.727 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 1/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2024-04-24T15:54:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2024-04-24T15:54:06.006 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2024-04-24T15:54:06.984 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 kB/s | 4.5 kB 00:00 2024-04-24T15:54:07.898 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-24T15:54:08.112 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:01 ago on Wed 24 Apr 2024 03:50:14 PM UTC. 2024-04-24T15:54:10.723 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:54:10.724 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================= 2024-04-24T15:54:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:54:10.724 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================= 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-24T15:54:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================================= 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-04-24T15:54:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:10.727 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.9 M 2024-04-24T15:54:10.727 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 25 M 2024-04-24T15:54:10.727 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:54:11.067 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 162 kB/s | 33 kB 00:00 2024-04-24T15:54:11.109 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 66 kB/s | 16 kB 00:00 2024-04-24T15:54:11.151 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): ceph-mgr-rook-17.2.7-915.g330067db.el8. 165 kB/s | 47 kB 00:00 2024-04-24T15:54:11.176 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 747 kB/s | 81 kB 00:00 2024-04-24T15:54:11.301 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 510 kB/s | 61 kB 00:00 2024-04-24T15:54:11.326 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-04-24T15:54:11.360 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 4.6 MB/s | 1.2 MB 00:00 2024-04-24T15:54:11.427 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-jsonpointer-1.10-11.el8.noarch. 159 kB/s | 20 kB 00:00 2024-04-24T15:54:11.561 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 471 kB/s | 110 kB 00:00 2024-04-24T15:54:11.586 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 127 kB/s | 27 kB 00:00 2024-04-24T15:54:11.687 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 488 kB/s | 126 kB 00:00 2024-04-24T15:54:11.854 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 530 kB/s | 155 kB 00:00 2024-04-24T15:54:11.870 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 150 kB/s | 43 kB 00:00 2024-04-24T15:54:11.873 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:11.873 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-04-24T15:54:11.950 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:54:11.969 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:54:11.969 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:54:12.063 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:54:12.073 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:54:12.746 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:54:12.880 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2024-04-24T15:54:12.880 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:54:12.973 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-24T15:54:13.227 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T15:54:13.447 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:13.448 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:54:13.499 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2024-04-24T15:54:13.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T15:54:13.784 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2024-04-24T15:54:13.987 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2024-04-24T15:54:14.258 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2024-04-24T15:54:14.439 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2024-04-24T15:54:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2024-04-24T15:54:14.883 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-24T15:54:15.152 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2024-04-24T15:54:15.854 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-24T15:54:16.351 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2024-04-24T15:54:16.541 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 20 kB/s | 4.1 kB 00:00 2024-04-24T15:54:16.654 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2024-04-24T15:54:16.849 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:17.152 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T15:54:17.152 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 1/13 2024-04-24T15:54:17.152 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2024-04-24T15:54:17.152 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2024-04-24T15:54:17.152 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2024-04-24T15:54:17.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2024-04-24T15:54:18.834 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:54:18.835 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================== 2024-04-24T15:54:18.835 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:54:18.835 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================== 2024-04-24T15:54:18.835 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================== 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout:Install 5 Packages 2024-04-24T15:54:18.836 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:18.837 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.0 M 2024-04-24T15:54:18.837 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-04-24T15:54:18.837 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:54:19.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2024-04-24T15:54:19.243 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:54:19.259 INFO:teuthology.orchestra.run.smithi086.stdout:(1/5): ceph-mgr-cephadm-17.2.7-915.g330067db.el 415 kB/s | 123 kB 00:00 2024-04-24T15:54:19.292 INFO:teuthology.orchestra.run.smithi086.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 182 kB/s | 60 kB 00:00 2024-04-24T15:54:19.485 INFO:teuthology.orchestra.run.smithi086.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.8 MB/s | 538 kB 00:00 2024-04-24T15:54:19.518 INFO:teuthology.orchestra.run.smithi086.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 856 kB/s | 476 kB 00:00 2024-04-24T15:54:19.610 INFO:teuthology.orchestra.run.smithi086.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-04-24T15:54:19.611 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:19.612 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.7 MB/s | 6.0 MB 00:00 2024-04-24T15:54:19.665 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:54:19.677 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:54:19.677 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:54:19.764 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:54:19.774 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T15:54:19.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:19.820 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:54:19.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T15:54:20.354 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:54:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-24T15:54:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-24T15:54:21.620 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:21.917 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-24T15:54:22.099 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-24T15:54:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:22.418 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:22.419 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 1/5 2024-04-24T15:54:22.419 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-24T15:54:22.419 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:22.419 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-24T15:54:22.762 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-24T15:54:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:54:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================== 2024-04-24T15:54:25.180 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:54:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================== 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================== 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:Install 5 Packages 2024-04-24T15:54:25.181 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 6.0 M 2024-04-24T15:54:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-04-24T15:54:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:54:25.649 INFO:teuthology.orchestra.run.smithi122.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 543 kB/s | 60 kB 00:00 2024-04-24T15:54:25.700 INFO:teuthology.orchestra.run.smithi122.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.9 MB/s | 476 kB 00:00 2024-04-24T15:54:25.858 INFO:teuthology.orchestra.run.smithi122.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2024-04-24T15:54:25.883 INFO:teuthology.orchestra.run.smithi122.stdout:(4/5): ceph-mgr-cephadm-17.2.7-915.g330067db.el 357 kB/s | 123 kB 00:00 2024-04-24T15:54:25.909 INFO:teuthology.orchestra.run.smithi122.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.5 MB/s | 538 kB 00:00 2024-04-24T15:54:25.909 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:25.910 INFO:teuthology.orchestra.run.smithi122.stdout:Total 8.2 MB/s | 6.0 MB 00:00 2024-04-24T15:54:25.964 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:54:25.976 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:54:25.976 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:54:26.064 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:54:26.072 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:54:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:54:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-24T15:54:27.659 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-24T15:54:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:28.309 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-24T15:54:28.385 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2024-04-24T15:54:28.385 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:54:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 1/5 2024-04-24T15:54:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-24T15:54:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:28.833 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-24T15:54:29.039 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T15:54:29.040 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T15:54:29.041 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:29.041 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:54:29.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T15:54:29.650 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:31.589 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-24T15:54:32.342 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-24T15:54:33.439 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-24T15:54:33.439 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:54:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:33.988 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:33.989 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:54:34.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-04-24T15:54:34.511 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:35.007 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:54:35.008 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================== 2024-04-24T15:54:35.008 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:54:35.008 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================== 2024-04-24T15:54:35.008 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:54:35.008 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================== 2024-04-24T15:54:35.009 INFO:teuthology.orchestra.run.smithi032.stdout:Install 5 Packages 2024-04-24T15:54:35.010 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:35.010 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.0 M 2024-04-24T15:54:35.010 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-04-24T15:54:35.010 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:54:35.413 INFO:teuthology.orchestra.run.smithi032.stdout:(1/5): ceph-mgr-cephadm-17.2.7-915.g330067db.el 569 kB/s | 123 kB 00:00 2024-04-24T15:54:35.538 INFO:teuthology.orchestra.run.smithi032.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 176 kB/s | 60 kB 00:00 2024-04-24T15:54:35.722 INFO:teuthology.orchestra.run.smithi032.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2024-04-24T15:54:35.757 INFO:teuthology.orchestra.run.smithi032.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.4 MB/s | 538 kB 00:00 2024-04-24T15:54:35.789 INFO:teuthology.orchestra.run.smithi032.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 803 kB/s | 476 kB 00:00 2024-04-24T15:54:35.791 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:35.791 INFO:teuthology.orchestra.run.smithi032.stdout:Total 7.6 MB/s | 6.0 MB 00:00 2024-04-24T15:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:54:35.866 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:54:35.866 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:54:35.966 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:54:35.975 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:54:36.251 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 4.5 kB 00:00 2024-04-24T15:54:36.852 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-24T15:54:37.055 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:54:37.438 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-24T15:54:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-24T15:54:38.057 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-24T15:54:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T15:54:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 1/5 2024-04-24T15:54:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-24T15:54:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-24T15:54:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-24T15:54:38.993 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-24T15:54:38.993 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:54:39.160 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T15:54:39.161 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:39.162 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 959 k 2024-04-24T15:54:39.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.1 M 2024-04-24T15:54:39.163 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:54:39.454 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 285 kB/s | 83 kB 00:00 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:39.544 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:54:39.562 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-17.2.7-915.g330067db.el8.x86_6 2.1 MB/s | 876 kB 00:00 2024-04-24T15:54:39.563 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:39.563 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.3 MB/s | 959 kB 00:00 2024-04-24T15:54:39.575 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:54:39.596 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:54:39.596 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:54:39.681 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:54:39.689 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:54:39.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-04-24T15:54:39.856 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:54:40.075 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:40.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-24T15:54:40.304 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:54:40.665 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:54:40.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/2 2024-04-24T15:54:41.938 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-24T15:54:42.618 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:54:45.017 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:54:45.018 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:54:45.018 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout: x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-24T15:54:45.019 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:54:45.020 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 959 k 2024-04-24T15:54:45.021 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 3.1 M 2024-04-24T15:54:45.021 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:54:45.221 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 412 kB/s | 83 kB 00:00 2024-04-24T15:54:45.304 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-17.2.7-915.g330067db.el8.x86_6 3.0 MB/s | 876 kB 00:00 2024-04-24T15:54:45.304 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:45.305 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.3 MB/s | 959 kB 00:00 2024-04-24T15:54:45.316 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:54:45.337 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:54:45.337 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:54:45.421 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:54:45.430 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:54:45.592 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:54:45.897 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-24T15:54:46.038 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:54:46.412 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:54:46.412 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/2 2024-04-24T15:54:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-24T15:54:52.470 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:54:52.936 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-24T15:54:52.936 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:53.098 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:54:53.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-04-24T15:54:53.477 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:53.478 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:54:53.478 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-24T15:54:53.478 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:53.478 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:54:53.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-04-24T15:54:53.713 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:54.010 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:54:55.632 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-24T15:54:55.890 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-24T15:54:56.386 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-24T15:54:56.539 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-24T15:54:58.869 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:54:58.870 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:54:58.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:54:58.871 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 267 k 2024-04-24T15:54:58.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.3 M 2024-04-24T15:54:58.871 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:54:59.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-volume-17.2.7-915.g330067db.el8.noarch.rpm 1.3 MB/s | 267 kB 00:00 2024-04-24T15:54:59.076 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:59.077 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.3 MB/s | 267 kB 00:00 2024-04-24T15:54:59.077 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:54:59.087 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:54:59.087 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:54:59.105 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:54:59.106 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:54:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:54:59.106 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:54:59.106 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:54:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-04-24T15:54:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:54:59.108 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 959 k 2024-04-24T15:54:59.108 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 3.1 M 2024-04-24T15:54:59.108 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:54:59.117 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:54:59.126 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:54:59.291 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 454 kB/s | 83 kB 00:00 2024-04-24T15:54:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:54:59.508 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-fuse-17.2.7-915.g330067db.el8.x86_6 2.1 MB/s | 876 kB 00:00 2024-04-24T15:54:59.509 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:54:59.509 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.3 MB/s | 959 kB 00:00 2024-04-24T15:54:59.525 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:54:59.549 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:54:59.549 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:54:59.588 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-24T15:54:59.589 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:54:59.651 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:54:59.661 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:54:59.753 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:54:59.891 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:55:00.141 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:00.141 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:55:00.141 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-24T15:55:00.141 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:00.141 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:55:00.156 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:00.187 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-24T15:55:00.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-04-24T15:55:00.402 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:55:00.688 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T15:55:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/2 2024-04-24T15:55:02.521 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-24T15:55:03.144 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-24T15:55:05.558 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:05.559 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:55:05.560 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:05.560 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 267 k 2024-04-24T15:55:05.560 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.3 M 2024-04-24T15:55:05.560 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:55:06.085 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-volume-17.2.7-915.g330067db.el8.noarch.rpm 509 kB/s | 267 kB 00:00 2024-04-24T15:55:06.086 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:06.086 INFO:teuthology.orchestra.run.smithi122.stdout:Total 508 kB/s | 267 kB 00:00 2024-04-24T15:55:06.087 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:55:06.096 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:55:06.097 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:55:06.126 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:55:06.135 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:55:06.342 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:55:06.578 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:06.968 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:10.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-24T15:55:10.300 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:55:10.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:10.970 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:55:10.970 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-24T15:55:10.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:10.970 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:55:11.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-volume 2024-04-24T15:55:11.582 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:13.236 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:13.236 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:55:13.333 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-24T15:55:13.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:13.802 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:55:13.802 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:55:13.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:13.802 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:55:13.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-04-24T15:55:14.088 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-24T15:55:14.315 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:16.321 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 kB/s | 4.5 kB 00:00 2024-04-24T15:55:16.850 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:55:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:16.852 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 267 k 2024-04-24T15:55:16.852 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.3 M 2024-04-24T15:55:16.852 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:55:16.969 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-24T15:55:17.597 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-volume-17.2.7-915.g330067db.el8.noarch.rpm 359 kB/s | 267 kB 00:00 2024-04-24T15:55:17.598 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:17.599 INFO:teuthology.orchestra.run.smithi032.stdout:Total 358 kB/s | 267 kB 00:00 2024-04-24T15:55:17.599 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:55:17.612 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:55:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:55:17.650 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:55:17.661 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:55:17.862 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:55:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:18.372 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:18.373 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:55:18.580 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:18.918 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:18.918 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:55:18.918 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:55:18.918 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:18.918 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:55:19.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-04-24T15:55:19.279 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:55:19.280 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-24T15:55:19.280 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 130 k 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 464 k 2024-04-24T15:55:19.281 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:55:19.404 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.7-915.g330067db.el8.x86_64. 248 kB/s | 130 kB 00:00 2024-04-24T15:55:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:Total 247 kB/s | 130 kB 00:00 2024-04-24T15:55:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:55:19.812 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:55:19.812 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:55:19.868 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:55:19.876 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:55:20.040 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:55:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:20.476 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:21.229 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-24T15:55:21.824 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-04-24T15:55:24.174 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:55:24.174 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:24.174 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 130 k 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 464 k 2024-04-24T15:55:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:55:24.924 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-17.2.7-915.g330067db.el8.x86_64. 173 kB/s | 130 kB 00:00 2024-04-24T15:55:24.925 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:24.925 INFO:teuthology.orchestra.run.smithi122.stdout:Total 173 kB/s | 130 kB 00:00 2024-04-24T15:55:24.925 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:55:24.931 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:55:24.931 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:55:24.992 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:55:25.000 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:55:25.180 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:55:25.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:25.614 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T15:55:26.708 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:55:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:55:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:55:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:55:27.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-04-24T15:55:27.918 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:29.694 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-24T15:55:30.411 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-24T15:55:32.683 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:32.683 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:55:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 130 k 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 464 k 2024-04-24T15:55:33.061 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:55:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:33.229 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:55:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:55:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:33.229 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:55:33.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-04-24T15:55:33.636 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-17.2.7-915.g330067db.el8.x86_64. 226 kB/s | 130 kB 00:00 2024-04-24T15:55:33.636 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:33.637 INFO:teuthology.orchestra.run.smithi032.stdout:Total 226 kB/s | 130 kB 00:00 2024-04-24T15:55:33.637 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:55:33.644 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:55:33.644 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:55:33.714 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:55:33.724 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:55:33.731 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:33.947 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:55:34.155 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:35.696 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:35.696 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:55:35.851 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 kB/s | 4.5 kB 00:00 2024-04-24T15:55:36.255 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:36.255 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:55:36.255 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:55:36.255 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:36.255 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:55:36.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-04-24T15:55:36.444 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-24T15:55:36.778 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:38.460 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-24T15:55:38.677 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:55:38.872 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:55:38.873 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:55:38.873 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:55:38.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-04-24T15:55:39.027 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 4.1 kB 00:00 2024-04-24T15:55:39.353 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:41.044 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-24T15:55:41.149 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:55:41.335 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:55:41.336 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:55:41.336 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:55:41.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-04-24T15:55:41.663 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:55:41.842 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-24T15:55:43.978 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:55:43.979 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:43.979 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 109 k 2024-04-24T15:55:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:55:44.201 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.7-915.g330067db.el8.x86_64 112 kB/s | 25 kB 00:00 2024-04-24T15:55:44.201 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:44.202 INFO:teuthology.orchestra.run.smithi086.stdout:Total 112 kB/s | 25 kB 00:00 2024-04-24T15:55:44.202 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:55:44.205 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:55:44.206 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:55:44.207 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:55:44.216 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:55:44.273 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:55:44.393 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-24T15:55:44.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:44.699 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:46.602 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:55:46.808 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:46.809 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 25 k 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 109 k 2024-04-24T15:55:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:55:46.964 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-17.2.7-915.g330067db.el8.x86_64 161 kB/s | 25 kB 00:00 2024-04-24T15:55:46.965 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:46.965 INFO:teuthology.orchestra.run.smithi122.stdout:Total 160 kB/s | 25 kB 00:00 2024-04-24T15:55:46.965 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:55:46.969 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:55:46.969 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:55:46.970 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:55:46.979 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:55:47.036 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:55:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:47.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:47.292 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:55:47.292 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:55:47.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:47.292 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:55:47.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-04-24T15:55:47.465 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:47.880 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:49.866 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-24T15:55:50.566 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-24T15:55:53.053 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:55:53.266 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:55:53.268 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:55:53.268 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:55:53.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-04-24T15:55:53.832 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:54.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:54.132 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:55:54.722 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:54.722 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:55:54.722 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:55:54.722 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:55:54.723 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:55:54.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-04-24T15:55:55.284 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:55.731 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-24T15:55:56.438 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-24T15:55:56.955 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 kB/s | 4.5 kB 00:00 2024-04-24T15:55:57.117 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:57.118 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:55:57.544 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-04-24T15:55:57.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:57.664 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:55:57.664 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:55:57.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:55:57.664 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:55:57.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-04-24T15:55:58.180 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:55:59.158 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:55:59.158 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:59.158 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-24T15:55:59.158 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:55:59.159 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 25 k 2024-04-24T15:55:59.160 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 109 k 2024-04-24T15:55:59.160 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:55:59.515 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-17.2.7-915.g330067db.el8.x86_64 70 kB/s | 25 kB 00:00 2024-04-24T15:55:59.516 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:55:59.516 INFO:teuthology.orchestra.run.smithi032.stdout:Total 70 kB/s | 25 kB 00:00 2024-04-24T15:55:59.516 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:55:59.522 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:55:59.522 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:55:59.525 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:55:59.535 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:55:59.604 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:55:59.774 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:55:59.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:55:59.965 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:55:59.966 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:55:59.966 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:55:59.993 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-24T15:56:00.061 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:00.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-04-24T15:56:00.471 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:00.652 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-24T15:56:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-24T15:56:02.788 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:56:02.976 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:02.977 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:02.977 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:03.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-04-24T15:56:03.473 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:56:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:05.235 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-24T15:56:05.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-04-24T15:56:05.727 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:05.890 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-24T15:56:07.608 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-24T15:56:08.031 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:08.228 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:08.229 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:08.229 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:08.277 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-24T15:56:08.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-04-24T15:56:08.730 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:10.422 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:10.617 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:10.618 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:56:10.618 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:10.649 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-24T15:56:10.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-04-24T15:56:11.114 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:11.283 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-24T15:56:11.400 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:11.400 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:56:12.035 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:56:12.035 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:56:12.035 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:56:12.035 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:56:12.036 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:12.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-04-24T15:56:12.649 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:13.013 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-24T15:56:13.403 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:13.597 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:13.598 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:13.598 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:13.604 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-24T15:56:13.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-04-24T15:56:14.101 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:14.539 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-24T15:56:15.277 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:56:15.722 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:15.912 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:15.913 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:56:15.913 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:16.029 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-24T15:56:16.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-04-24T15:56:16.409 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 21 kB/s | 4.1 kB 00:00 2024-04-24T15:56:17.730 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:17.945 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:17.946 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:17.946 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:18.077 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 63 kB/s | 4.5 kB 00:00 2024-04-24T15:56:18.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-04-24T15:56:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:18.652 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 kB/s | 4.1 kB 00:00 2024-04-24T15:56:18.870 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:19.058 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:19.059 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:19.059 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:19.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-04-24T15:56:19.564 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:20.279 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 kB/s | 4.5 kB 00:00 2024-04-24T15:56:20.786 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:20.977 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:20.978 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:56:20.978 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:21.043 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:56:21.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-04-24T15:56:21.229 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 kB/s | 4.5 kB 00:00 2024-04-24T15:56:21.479 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:21.835 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 kB/s | 4.1 kB 00:00 2024-04-24T15:56:23.317 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-24T15:56:23.401 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:23.613 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:23.614 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:23.614 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:23.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-04-24T15:56:23.949 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-24T15:56:23.991 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:24.176 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:24.181 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:24.182 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:24.183 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:24.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-04-24T15:56:24.676 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:25.885 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 kB/s | 4.5 kB 00:00 2024-04-24T15:56:26.109 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:26.300 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:26.302 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:56:26.302 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:26.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-04-24T15:56:26.506 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-24T15:56:26.558 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 71 kB/s | 4.1 kB 00:00 2024-04-24T15:56:26.798 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:27.171 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-24T15:56:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-24T15:56:29.055 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:29.056 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-24T15:56:29.273 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:29.275 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:29.275 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:29.310 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:29.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-04-24T15:56:29.498 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:29.499 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:56:29.499 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:29.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-04-24T15:56:29.841 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:29.992 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:31.490 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:31.490 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 91 k 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 268 k 2024-04-24T15:56:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:56:31.609 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-24T15:56:31.728 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-24T15:56:31.823 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.7-915.g330067db.el8.x86_64.rpm 273 kB/s | 91 kB 00:00 2024-04-24T15:56:31.823 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:31.824 INFO:teuthology.orchestra.run.smithi086.stdout:Total 272 kB/s | 91 kB 00:00 2024-04-24T15:56:31.824 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:56:31.834 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:56:31.834 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:56:31.893 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:56:31.902 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:56:32.018 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:56:32.214 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:32.295 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2024-04-24T15:56:32.347 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-04-24T15:56:32.479 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:34.752 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:34.769 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:56:34.770 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:34.771 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:56:34.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:34.771 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 91 k 2024-04-24T15:56:34.771 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 268 k 2024-04-24T15:56:34.771 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:56:34.938 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-17.2.7-915.g330067db.el8.x86_64.rpm 542 kB/s | 91 kB 00:00 2024-04-24T15:56:34.939 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:34.939 INFO:teuthology.orchestra.run.smithi122.stdout:Total 538 kB/s | 91 kB 00:00 2024-04-24T15:56:34.939 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:56:34.949 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:56:34.949 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:56:34.964 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:34.966 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:34.966 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:35.005 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:56:35.013 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:56:35.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-04-24T15:56:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:56:35.321 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:35.520 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:37.457 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-24T15:56:38.167 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-24T15:56:40.657 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:40.869 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:40.870 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:40.870 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:41.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-04-24T15:56:41.426 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:43.406 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-24T15:56:44.180 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-24T15:56:45.136 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:56:45.685 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:45.685 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:56:45.686 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:56:45.686 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:45.686 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:56:45.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:45.719 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:56:45.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-04-24T15:56:46.205 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:46.274 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:46.275 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:56:46.275 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:56:46.275 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:46.275 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:56:46.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-04-24T15:56:46.653 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:56:46.810 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:46.868 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:46.869 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:56:46.869 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:56:47.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-04-24T15:56:47.434 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:47.916 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 69 kB/s | 4.5 kB 00:00 2024-04-24T15:56:48.402 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 kB/s | 4.5 kB 00:00 2024-04-24T15:56:48.519 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:56:49.074 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-24T15:56:49.217 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-24T15:56:49.997 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-24T15:56:50.821 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:56:50.821 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:50.821 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:56:50.822 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:56:50.823 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.5 M 2024-04-24T15:56:50.823 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 16 M 2024-04-24T15:56:50.823 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:56:51.505 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.5 M 2024-04-24T15:56:51.506 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 16 M 2024-04-24T15:56:51.507 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:56:52.392 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.7-915.g330067db.el8.x86_64.rpm 2.2 MB/s | 3.5 MB 00:01 2024-04-24T15:56:52.393 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:52.393 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.2 MB/s | 3.5 MB 00:01 2024-04-24T15:56:52.394 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:56:52.407 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:56:52.407 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:56:52.479 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:56:52.488 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:56:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 91 k 2024-04-24T15:56:52.716 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 268 k 2024-04-24T15:56:52.717 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:56:52.934 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:56:53.130 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:53.170 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-17.2.7-915.g330067db.el8.x86_64.rpm 200 kB/s | 91 kB 00:00 2024-04-24T15:56:53.171 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:53.171 INFO:teuthology.orchestra.run.smithi032.stdout:Total 199 kB/s | 91 kB 00:00 2024-04-24T15:56:53.171 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:56:53.184 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:56:53.184 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:56:53.252 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:56:53.262 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:56:53.352 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-17.2.7-915.g330067db.el8.x86_64.rpm 1.9 MB/s | 3.5 MB 00:01 2024-04-24T15:56:53.353 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:53.354 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.9 MB/s | 3.5 MB 00:01 2024-04-24T15:56:53.354 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:56:53.366 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:56:53.367 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:56:53.386 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:56:53.438 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:56:53.447 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:56:53.533 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:53.899 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:56:53.955 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:54.073 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:56:54.475 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:57:04.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:57:04.692 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:04.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:57:04.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-04-24T15:57:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:05.491 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:05.491 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:57:05.600 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:05.600 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:57:06.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:06.124 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:57:06.124 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:06.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:06.124 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:57:06.174 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:06.174 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:57:06.174 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:06.174 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:06.175 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:57:06.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-04-24T15:57:06.353 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-04-24T15:57:06.713 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:06.713 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:07.122 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-24T15:57:07.723 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:57:08.688 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 23 kB/s | 4.5 kB 00:00 2024-04-24T15:57:08.741 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-24T15:57:09.345 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-24T15:57:09.453 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-24T15:57:10.023 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-04-24T15:57:10.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:10.025 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 175 k 2024-04-24T15:57:10.025 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 556 k 2024-04-24T15:57:10.025 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-04-24T15:57:10.321 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.7-915.g330067db.el8.x86_64.rpm 589 kB/s | 175 kB 00:00 2024-04-24T15:57:10.322 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:57:10.322 INFO:teuthology.orchestra.run.smithi086.stdout:Total 587 kB/s | 175 kB 00:00 2024-04-24T15:57:10.322 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-04-24T15:57:10.333 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-04-24T15:57:10.333 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-04-24T15:57:10.391 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-04-24T15:57:10.402 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-04-24T15:57:10.517 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-04-24T15:57:10.658 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:11.668 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:57:11.668 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-24T15:57:11.669 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:11.670 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 175 k 2024-04-24T15:57:11.670 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 556 k 2024-04-24T15:57:11.670 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-24T15:57:11.868 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-17.2.7-915.g330067db.el8.x86_64.rpm 884 kB/s | 175 kB 00:00 2024-04-24T15:57:11.868 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:57:11.868 INFO:teuthology.orchestra.run.smithi122.stdout:Total 879 kB/s | 175 kB 00:00 2024-04-24T15:57:11.868 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-24T15:57:11.879 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-24T15:57:11.879 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-24T15:57:11.934 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-24T15:57:11.943 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-24T15:57:12.062 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-24T15:57:12.173 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:57:12.173 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:12.173 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:57:12.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:12.175 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.5 M 2024-04-24T15:57:12.175 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 16 M 2024-04-24T15:57:12.175 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:57:12.260 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:12.516 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:12.618 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-17.2.7-915.g330067db.el8.x86_64.rpm 7.8 MB/s | 3.5 MB 00:00 2024-04-24T15:57:12.619 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:57:12.619 INFO:teuthology.orchestra.run.smithi032.stdout:Total 7.8 MB/s | 3.5 MB 00:00 2024-04-24T15:57:12.619 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:57:12.634 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:57:12.634 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:57:12.714 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:57:12.726 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:57:13.210 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:57:13.582 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:27.535 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:27.535 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2024-04-24T15:57:28.096 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:28.096 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-04-24T15:57:28.096 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:28.096 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:57:28.096 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:57:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:28.214 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2024-04-24T15:57:28.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-04-24T15:57:28.614 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:28.769 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:28.769 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-24T15:57:28.769 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:28.769 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:57:28.769 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:57:28.933 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-04-24T15:57:29.294 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:30.141 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:30.142 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:57:30.358 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-24T15:57:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:30.816 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:57:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:30.816 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:57:30.954 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-04-24T15:57:30.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-04-24T15:57:31.231 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-24T15:57:31.416 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:31.891 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-24T15:57:33.088 INFO:teuthology.orchestra.run.smithi086.stdout:Package rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:57:33.277 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-04-24T15:57:33.278 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-04-24T15:57:33.278 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-04-24T15:57:33.428 DEBUG:teuthology.parallel:result is None 2024-04-24T15:57:33.635 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 20 kB/s | 4.5 kB 00:00 2024-04-24T15:57:34.010 INFO:teuthology.orchestra.run.smithi122.stdout:Package rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:57:34.199 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-24T15:57:34.200 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-24T15:57:34.200 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-24T15:57:34.350 DEBUG:teuthology.parallel:result is None 2024-04-24T15:57:34.425 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 21 kB/s | 4.1 kB 00:00 2024-04-24T15:57:37.148 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-24T15:57:37.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:37.150 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 175 k 2024-04-24T15:57:37.150 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 556 k 2024-04-24T15:57:37.150 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-24T15:57:37.456 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-17.2.7-915.g330067db.el8.x86_64.rpm 572 kB/s | 175 kB 00:00 2024-04-24T15:57:37.456 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:57:37.457 INFO:teuthology.orchestra.run.smithi032.stdout:Total 569 kB/s | 175 kB 00:00 2024-04-24T15:57:37.457 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-24T15:57:37.471 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-24T15:57:37.471 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-24T15:57:37.541 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-24T15:57:37.550 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-24T15:57:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-24T15:57:37.831 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:38.138 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:50.848 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T15:57:50.848 INFO:teuthology.orchestra.run.smithi032.stdout:Installed products updated. 2024-04-24T15:57:51.476 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:51.476 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-24T15:57:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:57:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:57:51.477 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:57:51.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-04-24T15:57:52.082 INFO:teuthology.orchestra.run.smithi032.stdout:Updating Subscription Management repositories. 2024-04-24T15:57:53.841 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-24T15:57:54.564 INFO:teuthology.orchestra.run.smithi032.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-24T15:57:57.046 INFO:teuthology.orchestra.run.smithi032.stdout:Package rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T15:57:57.263 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-24T15:57:57.265 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-24T15:57:57.265 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-24T15:57:57.416 DEBUG:teuthology.parallel:result is None 2024-04-24T15:57:57.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:57:57.565 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T15:57:57.593 INFO:teuthology.orchestra.run.smithi032.stdout:17.2.7-915.g330067db.el8 2024-04-24T15:57:57.593 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T15:57:57.594 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T15:57:57.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:57:57.826 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T15:57:57.853 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.7-915.g330067db.el8 2024-04-24T15:57:57.853 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T15:57:57.853 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T15:57:57.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:57:58.001 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T15:57:58.028 INFO:teuthology.orchestra.run.smithi122.stdout:17.2.7-915.g330067db.el8 2024-04-24T15:57:58.029 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T15:57:58.029 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T15:57:58.031 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T15:57:58.031 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:57:58.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T15:57:58.067 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:57:58.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T15:57:58.112 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:57:58.112 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T15:57:58.158 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T15:57:58.158 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:57:58.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T15:57:58.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T15:57:58.268 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:57:58.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T15:57:58.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T15:57:58.369 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:57:58.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T15:57:58.402 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T15:57:58.474 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T15:57:58.475 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:57:58.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T15:57:58.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T15:57:58.584 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:57:58.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T15:57:58.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T15:57:58.685 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:57:58.685 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T15:57:58.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T15:57:58.791 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T15:57:58.791 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:57:58.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T15:57:58.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T15:57:58.901 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:57:58.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T15:57:58.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T15:57:59.004 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:57:59.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T15:57:59.037 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T15:57:59.110 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T15:57:59.192 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T15:57:59.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-04-24T15:57:59.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-04-24T15:57:59.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-04-24T15:57:59.230 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T15:57:59.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T15:57:59.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T15:57:59.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T15:57:59.302 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T15:57:59.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T15:57:59.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T15:57:59.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T15:57:59.387 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T15:57:59.388 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T15:57:59.388 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7671585', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbjG2d46Cc6vc2UmtV449lwgybDTTObm0+JMiC2GMj1r1w+0bfO0hVfPSwbiPlEWN/cYdSTL5QgYOrgnfVOpGM=', 'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM/i7Hr+2nm2cYmKiAcPHe5wxWBQwHIvdGs+S4GkzEJo/mo+rFUjEgQ5EC9apO6MB8RPlPugryuaoONVEAC2L5U=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaKRPvBPlQ9VkPpvDNyl6FNgnrz5pOCXNsDaKxJDM5HvfdIQlDprbmi1ENIFrhnMHIVeBZecsWWr4ayG32MqAo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_14:09:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T15:57:59.388 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T15:57:59.418 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T15:57:59.421 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T15:57:59.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:57:59.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:57:59.482 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:57:59.512 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:57:59.512 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T15:57:59.574 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T15:57:59.574 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 182529 Links: 1 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 15:57:13.779754369 +0000 2024-04-24T15:57:59.636 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 15:48:44.431036176 +0000 2024-04-24T15:57:59.637 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 15:48:44.431036176 +0000 2024-04-24T15:57:59.637 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T15:57:59.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T15:57:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T15:57:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T15:57:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000133532 s, 3.8 MB/s 2024-04-24T15:57:59.710 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T15:57:59.773 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 181049 Links: 1 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 15:57:13.779754369 +0000 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 15:48:44.828028176 +0000 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 15:48:44.828028176 +0000 2024-04-24T15:57:59.835 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T15:57:59.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T15:57:59.907 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T15:57:59.908 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T15:57:59.908 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000116734 s, 4.4 MB/s 2024-04-24T15:57:59.909 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T15:57:59.971 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 181869 Links: 1 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 15:57:13.779754369 +0000 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 15:48:45.197020739 +0000 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 15:48:45.197020739 +0000 2024-04-24T15:58:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T15:58:00.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T15:58:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T15:58:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T15:58:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000129792 s, 3.9 MB/s 2024-04-24T15:58:00.108 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T15:58:00.171 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-24T15:58:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T15:58:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:00.233 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 181910 Links: 1 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 15:57:13.779754369 +0000 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 15:48:45.558013464 +0000 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 15:48:45.558013464 +0000 2024-04-24T15:58:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T15:58:00.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T15:58:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T15:58:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T15:58:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.00012679 s, 4.0 MB/s 2024-04-24T15:58:00.308 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T15:58:00.370 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:00.370 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:00.371 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T15:58:00.392 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T15:58:00.392 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 175757 Links: 1 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-24 15:56:53.308540258 +0000 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-24 15:48:03.474260389 +0000 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-24 15:48:03.474260389 +0000 2024-04-24T15:58:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-24T15:58:00.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T15:58:00.522 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-24T15:58:00.523 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-24T15:58:00.523 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000285691 s, 1.8 MB/s 2024-04-24T15:58:00.524 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T15:58:00.585 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-04-24T15:58:00.645 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T15:58:00.645 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 176971 Links: 1 2024-04-24T15:58:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-24 15:56:53.308540258 +0000 2024-04-24T15:58:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-24 15:48:03.812253197 +0000 2024-04-24T15:58:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-24 15:48:03.812253197 +0000 2024-04-24T15:58:00.646 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-24T15:58:00.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T15:58:00.714 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-24T15:58:00.714 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-24T15:58:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000249514 s, 2.1 MB/s 2024-04-24T15:58:00.716 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T15:58:00.775 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 175882 Links: 1 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-24 15:56:53.309540236 +0000 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-24 15:48:04.152245962 +0000 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-24 15:48:04.152245962 +0000 2024-04-24T15:58:00.833 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-24T15:58:00.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T15:58:00.899 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-24T15:58:00.900 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-24T15:58:00.900 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000190702 s, 2.7 MB/s 2024-04-24T15:58:00.901 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T15:58:00.960 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 178022 Links: 1 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-24 15:56:53.309540236 +0000 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-24 15:48:04.486238855 +0000 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-24 15:48:04.486238855 +0000 2024-04-24T15:58:01.018 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-04-24T15:58:01.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T15:58:01.084 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-24T15:58:01.084 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-24T15:58:01.084 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000242011 s, 2.1 MB/s 2024-04-24T15:58:01.085 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T15:58:01.143 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:01.143 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:58:01.143 INFO:tasks.ceph:Generating config... 2024-04-24T15:58:01.144 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T15:58:01.145 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T15:58:01.145 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T15:58:01.145 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T15:58:01.145 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T15:58:01.145 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T15:58:01.145 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T15:58:01.145 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T15:58:01.145 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-24T15:58:01.146 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T15:58:01.146 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T15:58:01.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T15:58:01.201 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T15:58:01.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T15:58:01.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T15:58:01.286 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.c', '[v2:172.21.15.32:3301,v1:172.21.15.32:6790]'), ('mon.b', '172.21.15.86')] 2024-04-24T15:58:01.286 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32,[v2:172.21.15.32:3301,v1:172.21.15.32:6790],172.21.15.86', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-24T15:58:01.287 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:01.287 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:58:01.347 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --addv c '[v2:172.21.15.32:3301,v1:172.21.15.32:6790]' --add b 172.21.15.86 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = octopus 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:fsid c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-04-24T15:58:01.443249+0000 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-24T15:58:01.443249+0000 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 15 (octopus) 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-04-24T15:58:01.440 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-24T15:58:01.441 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-04-24T15:58:01.441 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0] mon.c 2024-04-24T15:58:01.441 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T15:58:01.442 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:58:01.461 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c4fc9996-5ff9-4e71-b213-9020967fae1a... 2024-04-24T15:58:01.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T15:58:01.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T15:58:01.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.553 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.554 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-04-24T15:58:01.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.32,[v2:172.21.15.32:3301,v1:172.21.15.32:6790],172.21.15.86" 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on pool no app = False 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-04-24T15:58:01.556 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.557 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = bitmap 2024-04-24T15:58:01.558 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 1/20 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 1/20 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 4/10 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: osd max markdown count = 1000 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-04-24T15:58:01.559 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-04-24T15:58:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:58:01.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-04-24T15:58:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-04-24T15:58:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-04-24T15:58:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-04-24T15:58:01.565 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.32,[v2:172.21.15.32:3301,v1:172.21.15.32:6790],172.21.15.86" 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-04-24T15:58:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-04-24T15:58:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-04-24T15:58:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: osd max markdown count = 1000 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-04-24T15:58:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-24T15:58:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-04-24T15:58:01.572 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-04-24T15:58:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-04-24T15:58:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-04-24T15:58:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,[v2:172.21.15.32:3301,v1:172.21.15.32:6790],172.21.15.86" 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: mon client directed command retry = 5 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on pool no app = False 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: ms inject socket failures = 5000 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-04-24T15:58:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-04-24T15:58:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: osd max markdown count = 1000 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-04-24T15:58:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:01.581 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-04-24T15:58:01.582 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-04-24T15:58:01.584 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T15:58:01.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T15:58:01.690 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T15:58:01.691 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:01.691 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T15:58:01.711 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:01.711 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T15:58:01.772 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T15:58:01.772 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:01.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T15:58:01.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T15:58:01.859 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:01.859 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:01.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T15:58:01.921 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:01.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T15:58:01.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T15:58:01.961 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:01.962 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:02.020 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-04-24T15:58:02.020 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:58:02.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T15:58:02.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T15:58:02.065 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:58:02.065 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:02.126 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T15:58:02.126 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T15:58:02.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T15:58:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T15:58:02.210 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T15:58:02.210 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T15:58:02.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T15:58:02.290 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T15:58:02.305 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T15:58:02.306 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:58:02.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T15:58:02.340 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:02.340 INFO:tasks.ceph:role: osd.0 2024-04-24T15:58:02.340 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T15:58:02.341 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:02.415 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:02.415 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:02.415 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:02.437 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-04-24T15:58:02.439 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-04-24T15:58:02.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T15:58:02.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T15:58:02.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T15:58:02.646 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:02.646 INFO:tasks.ceph:role: osd.1 2024-04-24T15:58:02.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T15:58:02.646 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T15:58:02.720 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:02.720 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:02.720 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:02.721 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:02.744 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-04-24T15:58:02.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-04-24T15:58:02.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T15:58:02.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T15:58:02.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T15:58:02.948 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:02.948 INFO:tasks.ceph:role: osd.2 2024-04-24T15:58:02.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T15:58:02.949 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:03.024 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:03.025 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:03.047 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-04-24T15:58:03.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-04-24T15:58:03.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T15:58:03.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T15:58:03.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:03.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T15:58:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T15:58:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T15:58:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T15:58:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T15:58:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.272+0000 7fb0968ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T15:58:03.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:03.796+0000 7fb0968ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:04.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:04.562+0000 7fb0968ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:05.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:05.324+0000 7fb0968ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:06.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T15:58:06.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.250+0000 7fba2cded540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.250+0000 7fba2cded540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.250+0000 7fba2cded540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.250+0000 7fba2cded540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.250+0000 7fba2cded540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T15:58:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.251+0000 7fba2cded540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T15:58:06.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:06.777+0000 7fba2cded540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:07.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:07.543+0000 7fba2cded540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:08.307+0000 7fba2cded540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:09.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T15:58:09.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T15:58:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.233+0000 7f8192b11540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T15:58:09.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:09.760+0000 7f8192b11540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:10.523+0000 7f8192b11540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T15:58:11.286+0000 7f8192b11540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:12.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T15:58:12.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T15:58:12.147 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:12.147 INFO:tasks.ceph:role: osd.3 2024-04-24T15:58:12.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T15:58:12.147 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:12.224 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:12.653 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-24T15:58:12.655 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-24T15:58:12.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-24T15:58:12.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T15:58:12.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T15:58:12.927 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:12.927 INFO:tasks.ceph:role: osd.4 2024-04-24T15:58:12.927 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T15:58:12.927 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:13.003 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:13.432 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-24T15:58:13.434 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-24T15:58:13.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-24T15:58:13.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T15:58:13.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T15:58:13.650 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:13.650 INFO:tasks.ceph:role: osd.5 2024-04-24T15:58:13.650 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T15:58:13.650 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T15:58:13.725 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:13.725 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:13.725 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:13.725 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:13.726 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:14.178 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-24T15:58:14.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-24T15:58:14.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-24T15:58:14.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T15:58:14.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:14.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.473+0000 7fe6ba2ae540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T15:58:14.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.476+0000 7fe6ba2ae540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T15:58:14.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.476+0000 7fe6ba2ae540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T15:58:14.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.476+0000 7fe6ba2ae540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T15:58:14.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.476+0000 7fe6ba2ae540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T15:58:14.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:14.480+0000 7fe6ba2ae540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T15:58:15.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:15.007+0000 7fe6ba2ae540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:15.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:15.767+0000 7fe6ba2ae540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:16.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:16.533+0000 7fe6ba2ae540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:17.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T15:58:17.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.483+0000 7f7f974d3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T15:58:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.484+0000 7f7f974d3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T15:58:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.484+0000 7f7f974d3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T15:58:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.484+0000 7f7f974d3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T15:58:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.484+0000 7f7f974d3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T15:58:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:17.484+0000 7f7f974d3540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T15:58:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:18.014+0000 7f7f974d3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:18.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:18.783+0000 7f7f974d3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:19.546+0000 7f7f974d3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:20.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T15:58:20.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.469+0000 7fc456108540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T15:58:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.469+0000 7fc456108540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T15:58:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.469+0000 7fc456108540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T15:58:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.469+0000 7fc456108540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T15:58:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.469+0000 7fc456108540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T15:58:20.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.470+0000 7fc456108540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T15:58:20.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:20.996+0000 7fc456108540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:21.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:21.759+0000 7fc456108540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:22.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-24T15:58:22.528+0000 7fc456108540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:23.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T15:58:23.355 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T15:58:23.356 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:23.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T15:58:23.390 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:23.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T15:58:23.465 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:23.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T15:58:23.539 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:23.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T15:58:23.571 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:23.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T15:58:23.640 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:23.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T15:58:23.708 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:23.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T15:58:23.775 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:58:23.775 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T15:58:23.797 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T15:58:23.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T15:58:23.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T15:58:23.829 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-04-24T15:58:23.829 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQALLClmb99MEBAAXOQU721iEbQHrsVBb0AcGw== 2024-04-24T15:58:23.829 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAOLClmeIkDDxAAmQ3pjFOQyvZA8MQ9Q8pljw== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQARLClmkk79DRAAFdZi83GWzl+Rrj+xVAIT5w== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAKLClmdZ6TDBAAc5nhNOfrAhpCXAlSZPb+LQ== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAWLClm1NBYHBAAC9GkUwZAauUuDOA7kqsAnQ== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAZLClmt7LwHBAAx/StZpqvGH54IqE2ho/zXA== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAcLClmTywDHBAAsofSb3HhChQe5HnFi1iUTA== 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-04-24T15:58:23.830 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAKLClm9Wd3ERAAHh4Bi9g7MB7eNCMuor/vSQ== 2024-04-24T15:58:23.840 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-04-24T15:58:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQALLClmb99MEBAAXOQU721iEbQHrsVBb0AcGw== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAOLClmeIkDDxAAmQ3pjFOQyvZA8MQ9Q8pljw== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQARLClmkk79DRAAFdZi83GWzl+Rrj+xVAIT5w== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAKLClmdZ6TDBAAc5nhNOfrAhpCXAlSZPb+LQ== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAWLClm1NBYHBAAC9GkUwZAauUuDOA7kqsAnQ== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAZLClmt7LwHBAAx/StZpqvGH54IqE2ho/zXA== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAcLClmTywDHBAAsofSb3HhChQe5HnFi1iUTA== 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-04-24T15:58:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAKLClm9Wd3ERAAHh4Bi9g7MB7eNCMuor/vSQ== 2024-04-24T15:58:23.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:23.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:23.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:23.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:23.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:23.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T15:58:24.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T15:58:24.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:24.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:24.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:24.333 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T15:58:24.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T15:58:24.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T15:58:24.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T15:58:24.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T15:58:24.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T15:58:24.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T15:58:24.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T15:58:24.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T15:58:24.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T15:58:24.862 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:24.864 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T15:58:24.919 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T15:58:24.919 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T15:58:24.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T15:58:24.925 INFO:tasks.ceph.mon.a:Started 2024-04-24T15:58:24.925 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T15:58:24.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T15:58:24.928 INFO:tasks.ceph.mon.c:Started 2024-04-24T15:58:24.928 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T15:58:24.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T15:58:24.962 INFO:tasks.ceph.mon.b:Started 2024-04-24T15:58:24.962 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T15:58:24.962 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T15:58:24.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T15:58:24.964 INFO:tasks.ceph.mgr.x:Started 2024-04-24T15:58:24.964 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T15:58:24.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T15:58:30.662 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:30.664+0000 7ffaf0cc0180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T15:58:30.738 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:30.741+0000 7ffaf0cc0180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T15:58:30.809 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:30.811+0000 7ffaf0cc0180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T15:58:30.876 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:30.879+0000 7ffaf0cc0180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T15:58:31.046 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.047+0000 7ffaf0cc0180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:58:31.188 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.190+0000 7ffaf0cc0180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T15:58:31.198 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-04-24T15:58:31.211 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T15:58:31.211 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:31.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T15:58:31.242 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:31.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T15:58:31.257 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.260+0000 7ffaf0cc0180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T15:58:31.316 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T15:58:31.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T15:58:31.389 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.392+0000 7ffaf0cc0180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T15:58:31.390 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:31.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T15:58:31.418 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:31.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T15:58:31.468 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.471+0000 7ffaf0cc0180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T15:58:31.487 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-24T15:58:31.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T15:58:31.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 51c39890-e79b-4030-a4a0-8c98ec2018ba 0 2024-04-24T15:58:31.637 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.639+0000 7ffaf0cc0180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T15:58:31.765 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:31.767+0000 7ffaf0cc0180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T15:58:31.819 INFO:teuthology.orchestra.run.smithi086.stdout:0 2024-04-24T15:58:31.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 1fb2523f-db36-46d9-80c4-dc08ac530daa 1 2024-04-24T15:58:32.084 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-04-24T15:58:32.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7d4dd13f-26d8-4c39-a6a6-020e91634f4e 2 2024-04-24T15:58:32.152 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:32.155+0000 7ffaf0cc0180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T15:58:32.310 INFO:teuthology.orchestra.run.smithi086.stdout:2 2024-04-24T15:58:32.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 03ed1eb6-8fa9-497c-b769-2743acc0e6aa 3 2024-04-24T15:58:32.377 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:32.379+0000 7ffaf0cc0180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T15:58:32.534 INFO:teuthology.orchestra.run.smithi086.stdout:3 2024-04-24T15:58:32.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new f7ff11ec-62d6-44f3-bea1-7ff62d280e2f 4 2024-04-24T15:58:32.616 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:32.618+0000 7ffaf0cc0180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T15:58:32.696 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:32.699+0000 7ffaf0cc0180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T15:58:32.760 INFO:teuthology.orchestra.run.smithi086.stdout:4 2024-04-24T15:58:32.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 423ca5ac-f6a3-496e-a4c5-9ac205e2e474 5 2024-04-24T15:58:32.988 INFO:teuthology.orchestra.run.smithi086.stdout:5 2024-04-24T15:58:33.001 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T15:58:33.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T15:58:33.003 INFO:tasks.ceph.osd.0:Started 2024-04-24T15:58:33.003 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T15:58:33.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T15:58:33.006 INFO:tasks.ceph.osd.1:Started 2024-04-24T15:58:33.006 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T15:58:33.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T15:58:33.008 INFO:tasks.ceph.osd.2:Started 2024-04-24T15:58:33.008 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T15:58:33.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T15:58:33.010 INFO:tasks.ceph.osd.3:Started 2024-04-24T15:58:33.010 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T15:58:33.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T15:58:33.012 INFO:tasks.ceph.osd.4:Started 2024-04-24T15:58:33.012 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T15:58:33.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T15:58:33.014 INFO:tasks.ceph.osd.5:Started 2024-04-24T15:58:33.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T15:58:33.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T15:58:32.987203+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:33.257 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T15:58:33.257 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T15:58:33.398 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:33.401+0000 7ffaf0cc0180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T15:58:33.480 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:33.482+0000 7ffaf0cc0180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T15:58:33.608 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:33.610+0000 7fdf793b1540 -1 Falling back to public interface 2024-04-24T15:58:33.609 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:58:33.610+0000 7fa3f27e6540 -1 Falling back to public interface 2024-04-24T15:58:33.611 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:33.613+0000 7f03fc494540 -1 Falling back to public interface 2024-04-24T15:58:33.611 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:33.614+0000 7ffaf0cc0180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T15:58:33.622 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:33.625+0000 7f840df96540 -1 Falling back to public interface 2024-04-24T15:58:33.626 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:33.629+0000 7f23ec2dc540 -1 Falling back to public interface 2024-04-24T15:58:33.628 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:33.631+0000 7f82adb27540 -1 Falling back to public interface 2024-04-24T15:58:33.862 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T15:58:34.074 INFO:teuthology.misc.health.smithi032.stdout: 2024-04-24T15:58:34.074 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":8,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T15:58:32.987203+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:34.086 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-24T15:58:34.167 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:34.170+0000 7ffaf0cc0180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T15:58:34.331 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:34.334+0000 7ffaf0cc0180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T15:58:34.496 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:34.498+0000 7ffaf0cc0180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T15:58:35.282 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:35.284+0000 7ffaf0cc0180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T15:58:35.496 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:35.498+0000 7ffaf0cc0180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:58:35.568 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:35.571+0000 7ffaf0cc0180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T15:58:35.728 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:35.731+0000 7ffaf0cc0180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T15:58:36.752 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:36.755+0000 7ffaaf776700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T15:58:36.753 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-24T15:58:36.756+0000 7ffaaf776700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T15:58:37.058 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:37.061+0000 7f840df96540 -1 osd.1 0 log_to_monitors true 2024-04-24T15:58:37.081 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:58:37.084+0000 7fa3f27e6540 -1 osd.5 0 log_to_monitors true 2024-04-24T15:58:37.109 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:37.112+0000 7f82adb27540 -1 osd.2 0 log_to_monitors true 2024-04-24T15:58:37.135 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:37.138+0000 7fdf793b1540 -1 osd.3 0 log_to_monitors true 2024-04-24T15:58:37.136 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:37.139+0000 7f23ec2dc540 -1 osd.0 0 log_to_monitors true 2024-04-24T15:58:37.139 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:37.142+0000 7f03fc494540 -1 osd.4 0 log_to_monitors true 2024-04-24T15:58:40.006 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:40.009+0000 7f82a4742700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T15:58:40.012 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:40.015+0000 7f829d539700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.032 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:40.035+0000 7f23e2ef7700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T15:58:40.036 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:40.039+0000 7f23dbcee700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.060 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:40.063+0000 7f8404bb1700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T15:58:40.065 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:40.067+0000 7f83fd9a8700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.403 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:40.405+0000 7f03f48b2700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T15:58:40.406 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:40.408+0000 7f03ebea6700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.410 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:58:40.412+0000 7fa3e9401700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T15:58:40.413 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:58:40.415+0000 7fa3e21f8700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.478 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:40.480+0000 7fdf6ffcc700 -1 osd.3 0 waiting for initial osdmap 2024-04-24T15:58:40.481 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:40.484+0000 7fdf68dc3700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:58:40.691 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T15:58:41.042 INFO:teuthology.misc.health.smithi032.stdout: 2024-04-24T15:58:41.042 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T15:58:41.011852+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:41.057 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-24T15:58:41.058 INFO:tasks.ceph:Creating RBD pool 2024-04-24T15:58:41.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-24T15:58:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'rbd' created 2024-04-24T15:58:42.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T15:58:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T15:58:43.048 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T15:58:43.049 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T15:58:43.049 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T15:58:43.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T15:58:43.049 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T15:58:43.485 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:43.500 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":4,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":41429},{"type":"v1","addr":"172.21.15.86:6825","nonce":41429}]},"active_addr":"172.21.15.86:6825/41429","active_change":"2024-04-24T15:58:35.745643+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1655033652}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1220991887}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2811142118}]}]} 2024-04-24T15:58:43.502 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T15:58:43.502 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T15:58:43.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T15:58:43.860 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:43.860 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T15:58:43.022113+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:43.876 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T15:58:43.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T15:58:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:44.243 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":14,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T15:58:44.026544+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:44.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T15:58:44.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T15:58:44.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T15:58:44.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T15:58:44.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T15:58:44.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T15:58:44.476 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:44.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T15:58:44.499 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:44.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T15:58:44.534 INFO:teuthology.orchestra.run.smithi032.stdout:47244640261 2024-04-24T15:58:44.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T15:58:44.555 INFO:teuthology.orchestra.run.smithi032.stdout:47244640261 2024-04-24T15:58:44.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T15:58:44.556 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:44.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T15:58:44.558 INFO:teuthology.orchestra.run.smithi032.stdout:47244640259 2024-04-24T15:58:44.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T15:58:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:47244640259 2024-04-24T15:58:44.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.4 2024-04-24T15:58:45.014 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:45.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.1 2024-04-24T15:58:45.046 INFO:teuthology.orchestra.run.smithi032.stdout:47244640259 2024-04-24T15:58:45.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-04-24T15:58:45.145 INFO:teuthology.orchestra.run.smithi032.stdout:47244640258 2024-04-24T15:58:45.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-04-24T15:58:45.168 INFO:teuthology.orchestra.run.smithi032.stdout:47244640259 2024-04-24T15:58:45.183 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:45.184 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-04-24T15:58:45.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2024-04-24T15:58:45.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T15:58:46.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T15:58:46.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T15:58:46.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T15:58:46.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T15:58:46.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T15:58:46.516 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:46.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.4 2024-04-24T15:58:46.539 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.580 INFO:teuthology.orchestra.run.smithi032.stdout:47244640261 2024-04-24T15:58:46.604 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-04-24T15:58:46.604 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.620 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:46.632 INFO:teuthology.orchestra.run.smithi032.stdout:47244640259 2024-04-24T15:58:46.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-04-24T15:58:46.636 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-04-24T15:58:46.649 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.669 INFO:teuthology.orchestra.run.smithi032.stdout:47244640261 2024-04-24T15:58:46.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2024-04-24T15:58:46.685 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.733 INFO:teuthology.orchestra.run.smithi032.stdout:47244640260 2024-04-24T15:58:46.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-04-24T15:58:46.749 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:46.749 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T15:58:46.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T15:58:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T15:58:47.114 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-24T15:58:45.751776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786064,"kb_used_data":1916,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444976,"statfs":{"total":579820584960,"available":577991655424,"internally_reserved":0,"allocated":1961984,"data_stored":1399166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.045068+0000","last_change":"2024-04-24T15:58:43.044292+0000","last_active":"2024-04-24T15:58:43.045068+0000","last_peered":"2024-04-24T15:58:43.045068+0000","last_clean":"2024-04-24T15:58:43.045068+0000","last_became_active":"2024-04-24T15:58:43.044019+0000","last_became_peered":"2024-04-24T15:58:43.044019+0000","last_unstale":"2024-04-24T15:58:43.045068+0000","last_undegraded":"2024-04-24T15:58:43.045068+0000","last_fullsized":"2024-04-24T15:58:43.045068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.032082+0000","last_change":"2024-04-24T15:58:43.030933+0000","last_active":"2024-04-24T15:58:43.032082+0000","last_peered":"2024-04-24T15:58:43.032082+0000","last_clean":"2024-04-24T15:58:43.032082+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T15:58:43.032082+0000","last_undegraded":"2024-04-24T15:58:43.032082+0000","last_fullsized":"2024-04-24T15:58:43.032082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.034763+0000","last_change":"2024-04-24T15:58:43.033676+0000","last_active":"2024-04-24T15:58:43.034763+0000","last_peered":"2024-04-24T15:58:43.034763+0000","last_clean":"2024-04-24T15:58:43.034763+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T15:58:43.034763+0000","last_undegraded":"2024-04-24T15:58:43.034763+0000","last_fullsized":"2024-04-24T15:58:43.034763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.032834+0000","last_change":"2024-04-24T15:58:43.031710+0000","last_active":"2024-04-24T15:58:43.032834+0000","last_peered":"2024-04-24T15:58:43.032834+0000","last_clean":"2024-04-24T15:58:43.032834+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T15:58:43.032834+0000","last_undegraded":"2024-04-24T15:58:43.032834+0000","last_fullsized":"2024-04-24T15:58:43.032834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.037428+0000","last_change":"2024-04-24T15:58:43.033207+0000","last_active":"2024-04-24T15:58:43.037428+0000","last_peered":"2024-04-24T15:58:43.037428+0000","last_clean":"2024-04-24T15:58:43.037428+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T15:58:43.037428+0000","last_undegraded":"2024-04-24T15:58:43.037428+0000","last_fullsized":"2024-04-24T15:58:43.037428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":159,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:44.039179+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T15:58:44.039179+0000","last_peered":"2024-04-24T15:58:44.039179+0000","last_clean":"2024-04-24T15:58:44.039179+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T15:58:44.039179+0000","last_undegraded":"2024-04-24T15:58:44.039179+0000","last_fullsized":"2024-04-24T15:58:44.039179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.034459+0000","last_change":"2024-04-24T15:58:43.033588+0000","last_active":"2024-04-24T15:58:43.034459+0000","last_peered":"2024-04-24T15:58:43.034459+0000","last_clean":"2024-04-24T15:58:43.034459+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T15:58:43.034459+0000","last_undegraded":"2024-04-24T15:58:43.034459+0000","last_fullsized":"2024-04-24T15:58:43.034459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.033569+0000","last_change":"2024-04-24T15:58:43.032704+0000","last_active":"2024-04-24T15:58:43.033569+0000","last_peered":"2024-04-24T15:58:43.033569+0000","last_clean":"2024-04-24T15:58:43.033569+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T15:58:43.033569+0000","last_undegraded":"2024-04-24T15:58:43.033569+0000","last_fullsized":"2024-04-24T15:58:43.033569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.045028+0000","last_change":"2024-04-24T15:58:43.044285+0000","last_active":"2024-04-24T15:58:43.045028+0000","last_peered":"2024-04-24T15:58:43.045028+0000","last_clean":"2024-04-24T15:58:43.045028+0000","last_became_active":"2024-04-24T15:58:43.044052+0000","last_became_peered":"2024-04-24T15:58:43.044052+0000","last_unstale":"2024-04-24T15:58:43.045028+0000","last_undegraded":"2024-04-24T15:58:43.045028+0000","last_fullsized":"2024-04-24T15:58:43.045028+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:13:47.283517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":629394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":135168,"data_stored":39026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":629394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:58:47.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T15:58:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:47.467 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T15:58:47.480 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-24T15:58:45.751776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786064,"kb_used_data":1916,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444976,"statfs":{"total":579820584960,"available":577991655424,"internally_reserved":0,"allocated":1961984,"data_stored":1399166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.045068+0000","last_change":"2024-04-24T15:58:43.044292+0000","last_active":"2024-04-24T15:58:43.045068+0000","last_peered":"2024-04-24T15:58:43.045068+0000","last_clean":"2024-04-24T15:58:43.045068+0000","last_became_active":"2024-04-24T15:58:43.044019+0000","last_became_peered":"2024-04-24T15:58:43.044019+0000","last_unstale":"2024-04-24T15:58:43.045068+0000","last_undegraded":"2024-04-24T15:58:43.045068+0000","last_fullsized":"2024-04-24T15:58:43.045068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.032082+0000","last_change":"2024-04-24T15:58:43.030933+0000","last_active":"2024-04-24T15:58:43.032082+0000","last_peered":"2024-04-24T15:58:43.032082+0000","last_clean":"2024-04-24T15:58:43.032082+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T15:58:43.032082+0000","last_undegraded":"2024-04-24T15:58:43.032082+0000","last_fullsized":"2024-04-24T15:58:43.032082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.034763+0000","last_change":"2024-04-24T15:58:43.033676+0000","last_active":"2024-04-24T15:58:43.034763+0000","last_peered":"2024-04-24T15:58:43.034763+0000","last_clean":"2024-04-24T15:58:43.034763+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T15:58:43.034763+0000","last_undegraded":"2024-04-24T15:58:43.034763+0000","last_fullsized":"2024-04-24T15:58:43.034763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.032834+0000","last_change":"2024-04-24T15:58:43.031710+0000","last_active":"2024-04-24T15:58:43.032834+0000","last_peered":"2024-04-24T15:58:43.032834+0000","last_clean":"2024-04-24T15:58:43.032834+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T15:58:43.032834+0000","last_undegraded":"2024-04-24T15:58:43.032834+0000","last_fullsized":"2024-04-24T15:58:43.032834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.037428+0000","last_change":"2024-04-24T15:58:43.033207+0000","last_active":"2024-04-24T15:58:43.037428+0000","last_peered":"2024-04-24T15:58:43.037428+0000","last_clean":"2024-04-24T15:58:43.037428+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T15:58:43.037428+0000","last_undegraded":"2024-04-24T15:58:43.037428+0000","last_fullsized":"2024-04-24T15:58:43.037428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":159,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:44.039179+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T15:58:44.039179+0000","last_peered":"2024-04-24T15:58:44.039179+0000","last_clean":"2024-04-24T15:58:44.039179+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T15:58:44.039179+0000","last_undegraded":"2024-04-24T15:58:44.039179+0000","last_fullsized":"2024-04-24T15:58:44.039179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.034459+0000","last_change":"2024-04-24T15:58:43.033588+0000","last_active":"2024-04-24T15:58:43.034459+0000","last_peered":"2024-04-24T15:58:43.034459+0000","last_clean":"2024-04-24T15:58:43.034459+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T15:58:43.034459+0000","last_undegraded":"2024-04-24T15:58:43.034459+0000","last_fullsized":"2024-04-24T15:58:43.034459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.033569+0000","last_change":"2024-04-24T15:58:43.032704+0000","last_active":"2024-04-24T15:58:43.033569+0000","last_peered":"2024-04-24T15:58:43.033569+0000","last_clean":"2024-04-24T15:58:43.033569+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T15:58:43.033569+0000","last_undegraded":"2024-04-24T15:58:43.033569+0000","last_fullsized":"2024-04-24T15:58:43.033569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T15:58:43.045028+0000","last_change":"2024-04-24T15:58:43.044285+0000","last_active":"2024-04-24T15:58:43.045028+0000","last_peered":"2024-04-24T15:58:43.045028+0000","last_clean":"2024-04-24T15:58:43.045028+0000","last_became_active":"2024-04-24T15:58:43.044052+0000","last_became_peered":"2024-04-24T15:58:43.044052+0000","last_unstale":"2024-04-24T15:58:43.045028+0000","last_undegraded":"2024-04-24T15:58:43.045028+0000","last_fullsized":"2024-04-24T15:58:43.045028+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:13:47.283517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":629394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":135168,"data_stored":39026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":122880,"data_stored":33784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":629394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:58:47.481 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T15:58:47.481 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T15:58:47.481 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T15:58:47.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T15:58:47.873 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:47.873 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T15:58:47.888 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T15:58:47.889 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T15:58:47.901 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T15:58:47.901 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi122.front.sepia.ceph.com 2024-04-24T15:58:47.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-04-24T15:58:49.056 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cache' created 2024-04-24T15:58:49.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2024-04-24T15:58:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2024-04-24T15:58:50.074 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-04-24T15:58:51.064 INFO:teuthology.orchestra.run.smithi122.stderr:set cache-mode for pool 'cache' to writeback 2024-04-24T15:58:51.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2024-04-24T15:58:52.067 INFO:teuthology.orchestra.run.smithi122.stderr:overlay for 'rbd' is now (or already was) 'cache' 2024-04-24T15:58:52.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-04-24T15:58:53.070 INFO:teuthology.orchestra.run.smithi122.stderr:set pool 3 hit_set_type to bloom 2024-04-24T15:58:53.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-04-24T15:58:54.078 INFO:teuthology.orchestra.run.smithi122.stderr:set pool 3 hit_set_count to 8 2024-04-24T15:58:54.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-04-24T15:58:55.080 INFO:teuthology.orchestra.run.smithi122.stderr:set pool 3 hit_set_period to 60 2024-04-24T15:58:55.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-04-24T15:58:56.087 INFO:teuthology.orchestra.run.smithi122.stderr:set pool 3 target_max_objects to 250 2024-04-24T15:58:56.100 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-24T15:58:56.111 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-24T15:58:56.111 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T15:58:56.111 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-24T15:58:56.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-24T15:58:56.497 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-24T15:58:56.498 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T15:58:56.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T15:58:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T15:58:56.862 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-24T15:58:56.085689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786284,"kb_used_data":2132,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444756,"statfs":{"total":579820584960,"available":577991430144,"internally_reserved":0,"allocated":2183168,"data_stored":1495679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020088"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.083971+0000","last_change":"2024-04-24T15:58:43.044292+0000","last_active":"2024-04-24T15:58:50.083971+0000","last_peered":"2024-04-24T15:58:50.083971+0000","last_clean":"2024-04-24T15:58:50.083971+0000","last_became_active":"2024-04-24T15:58:43.044019+0000","last_became_peered":"2024-04-24T15:58:43.044019+0000","last_unstale":"2024-04-24T15:58:50.083971+0000","last_undegraded":"2024-04-24T15:58:50.083971+0000","last_fullsized":"2024-04-24T15:58:50.083971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059263+0000","last_change":"2024-04-24T15:58:43.030933+0000","last_active":"2024-04-24T15:58:50.059263+0000","last_peered":"2024-04-24T15:58:50.059263+0000","last_clean":"2024-04-24T15:58:50.059263+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T15:58:50.059263+0000","last_undegraded":"2024-04-24T15:58:50.059263+0000","last_fullsized":"2024-04-24T15:58:50.059263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.066746+0000","last_change":"2024-04-24T15:58:50.066089+0000","last_active":"2024-04-24T15:58:50.066746+0000","last_peered":"2024-04-24T15:58:50.066746+0000","last_clean":"2024-04-24T15:58:50.066746+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T15:58:50.066746+0000","last_undegraded":"2024-04-24T15:58:50.066746+0000","last_fullsized":"2024-04-24T15:58:50.066746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:50:48.618571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.083927+0000","last_change":"2024-04-24T15:58:43.033676+0000","last_active":"2024-04-24T15:58:50.083927+0000","last_peered":"2024-04-24T15:58:50.083927+0000","last_clean":"2024-04-24T15:58:50.083927+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T15:58:50.083927+0000","last_undegraded":"2024-04-24T15:58:50.083927+0000","last_fullsized":"2024-04-24T15:58:50.083927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.088723+0000","last_change":"2024-04-24T15:58:50.088083+0000","last_active":"2024-04-24T15:58:50.088723+0000","last_peered":"2024-04-24T15:58:50.088723+0000","last_clean":"2024-04-24T15:58:50.088723+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T15:58:50.088723+0000","last_undegraded":"2024-04-24T15:58:50.088723+0000","last_fullsized":"2024-04-24T15:58:50.088723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:09:05.226724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059269+0000","last_change":"2024-04-24T15:58:43.031710+0000","last_active":"2024-04-24T15:58:50.059269+0000","last_peered":"2024-04-24T15:58:50.059269+0000","last_clean":"2024-04-24T15:58:50.059269+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T15:58:50.059269+0000","last_undegraded":"2024-04-24T15:58:50.059269+0000","last_fullsized":"2024-04-24T15:58:50.059269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.242253+0000","last_change":"2024-04-24T15:58:50.241646+0000","last_active":"2024-04-24T15:58:50.242253+0000","last_peered":"2024-04-24T15:58:50.242253+0000","last_clean":"2024-04-24T15:58:50.242253+0000","last_became_active":"2024-04-24T15:58:50.241320+0000","last_became_peered":"2024-04-24T15:58:50.241320+0000","last_unstale":"2024-04-24T15:58:50.242253+0000","last_undegraded":"2024-04-24T15:58:50.242253+0000","last_fullsized":"2024-04-24T15:58:50.242253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:13.855133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.062440+0000","last_change":"2024-04-24T15:58:43.033207+0000","last_active":"2024-04-24T15:58:50.062440+0000","last_peered":"2024-04-24T15:58:50.062440+0000","last_clean":"2024-04-24T15:58:50.062440+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T15:58:50.062440+0000","last_undegraded":"2024-04-24T15:58:50.062440+0000","last_fullsized":"2024-04-24T15:58:50.062440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.067336+0000","last_change":"2024-04-24T15:58:50.066620+0000","last_active":"2024-04-24T15:58:50.067336+0000","last_peered":"2024-04-24T15:58:50.067336+0000","last_clean":"2024-04-24T15:58:50.067336+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T15:58:50.067336+0000","last_undegraded":"2024-04-24T15:58:50.067336+0000","last_fullsized":"2024-04-24T15:58:50.067336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:25.629857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059281+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T15:58:50.059281+0000","last_peered":"2024-04-24T15:58:50.059281+0000","last_clean":"2024-04-24T15:58:50.059281+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T15:58:50.059281+0000","last_undegraded":"2024-04-24T15:58:50.059281+0000","last_fullsized":"2024-04-24T15:58:50.059281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059567+0000","last_change":"2024-04-24T15:58:43.033588+0000","last_active":"2024-04-24T15:58:50.059567+0000","last_peered":"2024-04-24T15:58:50.059567+0000","last_clean":"2024-04-24T15:58:50.059567+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T15:58:50.059567+0000","last_undegraded":"2024-04-24T15:58:50.059567+0000","last_fullsized":"2024-04-24T15:58:50.059567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059430+0000","last_change":"2024-04-24T15:58:43.032704+0000","last_active":"2024-04-24T15:58:50.059430+0000","last_peered":"2024-04-24T15:58:50.059430+0000","last_clean":"2024-04-24T15:58:50.059430+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T15:58:50.059430+0000","last_undegraded":"2024-04-24T15:58:50.059430+0000","last_fullsized":"2024-04-24T15:58:50.059430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.062494+0000","last_change":"2024-04-24T15:58:43.044285+0000","last_active":"2024-04-24T15:58:50.062494+0000","last_peered":"2024-04-24T15:58:50.062494+0000","last_clean":"2024-04-24T15:58:50.062494+0000","last_became_active":"2024-04-24T15:58:43.044052+0000","last_became_peered":"2024-04-24T15:58:43.044052+0000","last_unstale":"2024-04-24T15:58:50.062494+0000","last_undegraded":"2024-04-24T15:58:50.062494+0000","last_fullsized":"2024-04-24T15:58:50.062494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:13:47.283517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298104,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073736,"statfs":{"total":96636764160,"available":96331505664,"internally_reserved":0,"allocated":765952,"data_stored":645476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":172032,"data_stored":55108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298104,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073736,"statfs":{"total":96636764160,"available":96331505664,"internally_reserved":0,"allocated":765952,"data_stored":645476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:58:56.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T15:58:57.213 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:58:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T15:58:57.228 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-24T15:58:56.085689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786284,"kb_used_data":2132,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444756,"statfs":{"total":579820584960,"available":577991430144,"internally_reserved":0,"allocated":2183168,"data_stored":1495679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020088"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.083971+0000","last_change":"2024-04-24T15:58:43.044292+0000","last_active":"2024-04-24T15:58:50.083971+0000","last_peered":"2024-04-24T15:58:50.083971+0000","last_clean":"2024-04-24T15:58:50.083971+0000","last_became_active":"2024-04-24T15:58:43.044019+0000","last_became_peered":"2024-04-24T15:58:43.044019+0000","last_unstale":"2024-04-24T15:58:50.083971+0000","last_undegraded":"2024-04-24T15:58:50.083971+0000","last_fullsized":"2024-04-24T15:58:50.083971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059263+0000","last_change":"2024-04-24T15:58:43.030933+0000","last_active":"2024-04-24T15:58:50.059263+0000","last_peered":"2024-04-24T15:58:50.059263+0000","last_clean":"2024-04-24T15:58:50.059263+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T15:58:50.059263+0000","last_undegraded":"2024-04-24T15:58:50.059263+0000","last_fullsized":"2024-04-24T15:58:50.059263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.066746+0000","last_change":"2024-04-24T15:58:50.066089+0000","last_active":"2024-04-24T15:58:50.066746+0000","last_peered":"2024-04-24T15:58:50.066746+0000","last_clean":"2024-04-24T15:58:50.066746+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T15:58:50.066746+0000","last_undegraded":"2024-04-24T15:58:50.066746+0000","last_fullsized":"2024-04-24T15:58:50.066746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:50:48.618571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.083927+0000","last_change":"2024-04-24T15:58:43.033676+0000","last_active":"2024-04-24T15:58:50.083927+0000","last_peered":"2024-04-24T15:58:50.083927+0000","last_clean":"2024-04-24T15:58:50.083927+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T15:58:50.083927+0000","last_undegraded":"2024-04-24T15:58:50.083927+0000","last_fullsized":"2024-04-24T15:58:50.083927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.088723+0000","last_change":"2024-04-24T15:58:50.088083+0000","last_active":"2024-04-24T15:58:50.088723+0000","last_peered":"2024-04-24T15:58:50.088723+0000","last_clean":"2024-04-24T15:58:50.088723+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T15:58:50.088723+0000","last_undegraded":"2024-04-24T15:58:50.088723+0000","last_fullsized":"2024-04-24T15:58:50.088723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:09:05.226724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059269+0000","last_change":"2024-04-24T15:58:43.031710+0000","last_active":"2024-04-24T15:58:50.059269+0000","last_peered":"2024-04-24T15:58:50.059269+0000","last_clean":"2024-04-24T15:58:50.059269+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T15:58:50.059269+0000","last_undegraded":"2024-04-24T15:58:50.059269+0000","last_fullsized":"2024-04-24T15:58:50.059269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.242253+0000","last_change":"2024-04-24T15:58:50.241646+0000","last_active":"2024-04-24T15:58:50.242253+0000","last_peered":"2024-04-24T15:58:50.242253+0000","last_clean":"2024-04-24T15:58:50.242253+0000","last_became_active":"2024-04-24T15:58:50.241320+0000","last_became_peered":"2024-04-24T15:58:50.241320+0000","last_unstale":"2024-04-24T15:58:50.242253+0000","last_undegraded":"2024-04-24T15:58:50.242253+0000","last_fullsized":"2024-04-24T15:58:50.242253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:13.855133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.062440+0000","last_change":"2024-04-24T15:58:43.033207+0000","last_active":"2024-04-24T15:58:50.062440+0000","last_peered":"2024-04-24T15:58:50.062440+0000","last_clean":"2024-04-24T15:58:50.062440+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T15:58:50.062440+0000","last_undegraded":"2024-04-24T15:58:50.062440+0000","last_fullsized":"2024-04-24T15:58:50.062440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.067336+0000","last_change":"2024-04-24T15:58:50.066620+0000","last_active":"2024-04-24T15:58:50.067336+0000","last_peered":"2024-04-24T15:58:50.067336+0000","last_clean":"2024-04-24T15:58:50.067336+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T15:58:50.067336+0000","last_undegraded":"2024-04-24T15:58:50.067336+0000","last_fullsized":"2024-04-24T15:58:50.067336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:25.629857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059281+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T15:58:50.059281+0000","last_peered":"2024-04-24T15:58:50.059281+0000","last_clean":"2024-04-24T15:58:50.059281+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T15:58:50.059281+0000","last_undegraded":"2024-04-24T15:58:50.059281+0000","last_fullsized":"2024-04-24T15:58:50.059281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059567+0000","last_change":"2024-04-24T15:58:43.033588+0000","last_active":"2024-04-24T15:58:50.059567+0000","last_peered":"2024-04-24T15:58:50.059567+0000","last_clean":"2024-04-24T15:58:50.059567+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T15:58:50.059567+0000","last_undegraded":"2024-04-24T15:58:50.059567+0000","last_fullsized":"2024-04-24T15:58:50.059567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.059430+0000","last_change":"2024-04-24T15:58:43.032704+0000","last_active":"2024-04-24T15:58:50.059430+0000","last_peered":"2024-04-24T15:58:50.059430+0000","last_clean":"2024-04-24T15:58:50.059430+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T15:58:50.059430+0000","last_undegraded":"2024-04-24T15:58:50.059430+0000","last_fullsized":"2024-04-24T15:58:50.059430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T15:58:50.062494+0000","last_change":"2024-04-24T15:58:43.044285+0000","last_active":"2024-04-24T15:58:50.062494+0000","last_peered":"2024-04-24T15:58:50.062494+0000","last_clean":"2024-04-24T15:58:50.062494+0000","last_became_active":"2024-04-24T15:58:43.044052+0000","last_became_peered":"2024-04-24T15:58:43.044052+0000","last_unstale":"2024-04-24T15:58:50.062494+0000","last_undegraded":"2024-04-24T15:58:50.062494+0000","last_fullsized":"2024-04-24T15:58:50.062494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:13:47.283517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298104,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073736,"statfs":{"total":96636764160,"available":96331505664,"internally_reserved":0,"allocated":765952,"data_stored":645476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":172032,"data_stored":55108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":49873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298104,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073736,"statfs":{"total":96636764160,"available":96331505664,"internally_reserved":0,"allocated":765952,"data_stored":645476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:58:57.229 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T15:58:57.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-24T15:58:57.579 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-24T15:58:57.579 INFO:teuthology.orchestra.run.smithi032.stdout:fsid c4fc9996-5ff9-4e71-b213-9020967fae1a 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-24T15:58:30.292182+0000 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-24T15:58:56.080637+0000 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:crush_version 4 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:full_ratio 0.95 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:backfillfull_ratio 0.9 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:nearfull_ratio 0.85 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:require_min_compat_client luminous 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:min_compat_client jewel 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:require_osd_release quincy 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:stretch_mode_enabled false 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 lfor 18/18/18 flags hashpspool tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:pool 3 'cache' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 22 lfor 18/18/18 flags hashpspool,incomplete_clones tier_of 1 cache_mode writeback target_objects 250 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 60s x8 decay_rate 0 search_last_n 0 stripe_width 0 application rbd 2024-04-24T15:58:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:max_osd 6 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6808/41951,v1:172.21.15.32:6809/41951] [v2:172.21.15.32:6810/41951,v1:172.21.15.32:6811/41951] exists,up 51c39890-e79b-4030-a4a0-8c98ec2018ba 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] [v2:172.21.15.32:6802/41952,v1:172.21.15.32:6803/41952] exists,up 1fb2523f-db36-46d9-80c4-dc08ac530daa 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6812/41953,v1:172.21.15.32:6814/41953] [v2:172.21.15.32:6816/41953,v1:172.21.15.32:6818/41953] exists,up 7d4dd13f-26d8-4c39-a6a6-020e91634f4e 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6801/41923,v1:172.21.15.86:6803/41923] [v2:172.21.15.86:6805/41923,v1:172.21.15.86:6807/41923] exists,up 03ed1eb6-8fa9-497c-b769-2743acc0e6aa 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6816/41924,v1:172.21.15.86:6817/41924] [v2:172.21.15.86:6818/41924,v1:172.21.15.86:6819/41924] exists,up f7ff11ec-62d6-44f3-bea1-7ff62d280e2f 2024-04-24T15:58:57.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6800/41925,v1:172.21.15.86:6802/41925] [v2:172.21.15.86:6804/41925,v1:172.21.15.86:6806/41925] exists,up 423ca5ac-f6a3-496e-a4c5-9ac205e2e474 2024-04-24T15:58:57.595 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6808/41951,v1:172.21.15.32:6809/41951] [v2:172.21.15.32:6810/41951,v1:172.21.15.32:6811/41951] exists,up 51c39890-e79b-4030-a4a0-8c98ec2018ba', 'osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] [v2:172.21.15.32:6802/41952,v1:172.21.15.32:6803/41952] exists,up 1fb2523f-db36-46d9-80c4-dc08ac530daa', 'osd.2 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6812/41953,v1:172.21.15.32:6814/41953] [v2:172.21.15.32:6816/41953,v1:172.21.15.32:6818/41953] exists,up 7d4dd13f-26d8-4c39-a6a6-020e91634f4e', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6801/41923,v1:172.21.15.86:6803/41923] [v2:172.21.15.86:6805/41923,v1:172.21.15.86:6807/41923] exists,up 03ed1eb6-8fa9-497c-b769-2743acc0e6aa', 'osd.4 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6816/41924,v1:172.21.15.86:6817/41924] [v2:172.21.15.86:6818/41924,v1:172.21.15.86:6819/41924] exists,up f7ff11ec-62d6-44f3-bea1-7ff62d280e2f', 'osd.5 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6800/41925,v1:172.21.15.86:6802/41925] [v2:172.21.15.86:6804/41925,v1:172.21.15.86:6806/41925] exists,up 423ca5ac-f6a3-496e-a4c5-9ac205e2e474'] 2024-04-24T15:58:57.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mon.a", 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster": "ceph", 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout: "admin_socket_mode": "", 2024-04-24T15:58:57.728 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_client_required": "cephx, none", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_cluster_required": "cephx", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_debug": "true", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_service_required": "cephx", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_supported": "cephx", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio": "true", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_poll_ms": "250", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_reap_max": "16", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_async_discard": "false", 2024-04-24T15:58:57.729 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_block_size": "4096", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_inflight_ios": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_enable_discard": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_flock_retry": "3", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_inject_crash": "0", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring_hipri": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-24T15:58:57.730 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_type": "", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_alloc_size": "1048576", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_allocator": "hybrid", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_buffered_io": "true", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_for_zeros": "false", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_compact_log_sync": "false", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_max_log_runway": "4194304", 2024-04-24T15:58:57.731 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_max_prefetch": "1048576", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_min_flush_size": "524288", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_min_log_runway": "1048576", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_replay_recovery": "false", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_sync_write": "false", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_allocation_from_file": "true", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_allocator": "hybrid", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-24T15:58:57.732 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_create": "true", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_create": "false", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_path": "", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_size": "0", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_path": "", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_preallocate_file": "false", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_size": "107374182400", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_create": "false", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_path": "", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_size": "100663296", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs": "true", 2024-04-24T15:58:57.733 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_autotune": "true", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size": "0", 2024-04-24T15:58:57.734 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_type": "2q", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_clone_cow": "true", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-24T15:58:57.735 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_mode": "none", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_csum_type": "crc32c", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_freelist": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_bug21040": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-24T15:58:57.736 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_small_allocations": "0", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_default_buffered_read": "true", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_default_buffered_write": "false", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-24T15:58:57.737 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fail_eio": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mount": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_umount": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-24T15:58:57.738 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_ignore_data_csum": "false", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_op_age": "5.000000", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_alloc_size": "0", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size": "0", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-24T15:58:57.739 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_deferred_txc": "32", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size": "0", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_nid_prealloc": "1024", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_retry_disk_reads": "3", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_cf": "true", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-24T15:58:57.740 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_mem": "512", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_tracing": "false", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-24T15:58:57.741 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_zero_block_detection": "false", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-24T15:58:57.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_cluster_require_signatures": "false", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_cluster_require_version": "2", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_require_signatures": "false", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_require_version": "2", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_service_require_signatures": "false", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_service_require_version": "2", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_sign_messages": "true", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "chdir": "", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client_acl_type": "", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client_asio_thread_count": "2", 2024-04-24T15:58:57.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client_cache_mid": "0.750000", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_cache_size": "16384", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_caps_release_delay": "5", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_check_pool_perm": "true", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_force_sync_read": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_getattr_caps": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_inject_tick_delay": "0", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_die_on_failed_remount": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_dirsize_rbytes": "true", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_force_lazyio": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_fs": "", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_inject_release_failure": "false", 2024-04-24T15:58:57.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_max_inline_size": "4096", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mds_namespace": "", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_metadata": "", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_gid": "-1", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_timeout": "300", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_uid": "-1", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mountpoint": "/", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_notify_timeout": "10", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc": "true", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_dirty": "104857600", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_objects": "1000", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_size": "209715200", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_target_dirty": "8388608", 2024-04-24T15:58:57.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_permissions": "true", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_quota": "true", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_quota_df": "true", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_max_bytes": "0", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_max_periods": "4", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_min": "131072", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_reconnect_stale": "false", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_shutdown_timeout": "30", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_snapdir": ".snap", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_tick_interval": "1", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_trace": "", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_try_dentry_invalidate": "false", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_use_faked_inos": "false", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_use_random_mds": "false", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog": "false", 2024-04-24T15:58:57.746 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog_port": "12201", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_monitors": "default=true", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog": "false", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog_level": "info", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "-", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_network": "", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_network_interface": "", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_isal": "false", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_level": "5", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_winsize": "-15", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zstd_level": "1", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-24T15:58:57.747 INFO:teuthology.orchestra.run.smithi032.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_alien_op_num_threads": "6", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_osd_obc_lru_size": "10", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location": "", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location_hook": "", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location_hook_timeout": "10", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "daemonize": "false", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_alienstore": "0/5", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_allow_any_pool_priority": "false", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asok": "1/5", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asok_assert_abort": "false", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asserts_on_shutdown": "true", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_auth": "20/20", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bdev": "1/3", 2024-04-24T15:58:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bluefs": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bluestore": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_buffer": "0/1", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_ceph_exporter": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_cephfs_mirror": "0/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_cephsqlite": "0/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_civetweb": "1/10", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_client": "0/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_compressor": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_context": "0/1", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_crush": "1/1", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_crypto": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_deliberately_leak_memory": "false", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_disable_randomized_ping": "false", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_dpdk": "1/5", 2024-04-24T15:58:57.749 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_eventtrace": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_filer": "0/1", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_filestore": "1/3", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_finisher": "1/1", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_fuse": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_heartbeat_testing_span": "0", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_heartbeatmap": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_javaclient": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_journal": "1/3", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_journaler": "0/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_kstore": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_leveldb": "4/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_lockdep": "0/1", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mclock": "1/5", 2024-04-24T15:58:57.750 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_balancer": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_locker": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_log": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_log_expire": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_migrator": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_memdb": "4/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mgr": "2/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mgrc": "1/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mon": "20/20", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_monc": "0/10", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_ms": "1/1", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_none": "0/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objclass": "0/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objectcacher": "0/5", 2024-04-24T15:58:57.751 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objecter": "0/1", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_optracker": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_osd": "1/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_paxos": "20/20", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_perfcounter": "1/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_prioritycache": "1/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rados": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_mirror": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_pwl": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_replay": "0/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_refs": "0/0", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_reserver": "1/1", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw": "1/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_datacache": "1/5", 2024-04-24T15:58:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_sync": "1/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rocksdb": "4/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_cache": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_cleaner": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_device": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_journal": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_lba": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_odata": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_omap": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_onode": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_tm": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_striper": "0/1", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_test": "0/5", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_throttle": "1/1", 2024-04-24T15:58:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_timer": "0/1", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_tp": "0/5", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "device_failure_prediction_mode": "none", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_graylog": "false", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_journald": "false", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_stderr": "true", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_syslog": "false", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event_tracing": "false", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_addr": "0.0.0.0", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_http_port": "9926", 2024-04-24T15:58:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_prio_limit": "5", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_sort_metrics": "true", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_stats_period": "5", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "fake_statfs_for_testing": "0", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "fatal_signal_handlers": "true", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filer_max_purge_ops": "10", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filer_max_truncate_ops": "128", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_apply_finisher_threads": "1", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_blackhole": "false", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_btrfs_clone_range": "true", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_btrfs_snap": "true", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_caller_concurrency": "10", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_collect_device_partition_information": "true", 2024-04-24T15:58:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_commit_timeout": "600.000000", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_inject_read_err": "false", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_omap_check": "false", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_verify_split": "false", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_dump_file": "", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fadvise": "true", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fail_eio": "true", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fd_cache_shards": "16", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fd_cache_size": "128", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fiemap": "false", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fiemap_threshold": "4096", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-24T15:58:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_inject_stall": "0", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_parallel": "false", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_trailing": "false", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_writeahead": "false", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_kill_at": "0", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs": "0", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-24T15:58:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size": "0", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_merge_threshold": "-10", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_odsync_write": "false", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_backend": "rocksdb", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_backend_path": "", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_thread_timeout": "60", 2024-04-24T15:58:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_threads": "2", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_punch_hole": "false", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_ops": "50", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_seek_data_hole": "false", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_sloppy_crc": "false", 2024-04-24T15:58:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_splice": "false", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_split_multiple": "2", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_split_rand_factor": "20", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_update_to": "1000", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_enable": "true", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-24T15:58:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_xfs_extsize": "false", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_zfs_snap": "false", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fio_dir": "/tmp/fio", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c4fc9996-5ff9-4e71-b213-9020967fae1a", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_allow_other": "true", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_atomic_o_trunc": "true", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_big_writes": "true", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_debug": "false", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_default_permissions": "false", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_disable_pagecache": "false", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_max_write": "0", 2024-04-24T15:58:57.761 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_multithreaded": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_require_active_mds": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_set_user_groups": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_move": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_read": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_write": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_use_invalidate_cb": "true", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "gss_target_name": "ceph", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_file": "", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_inject_failure": "0", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_interval": "5", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "host": "smithi032", 2024-04-24T15:58:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "inject_early_sigterm": "false", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "jaeger_tracing_enable": "false", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_aio": "true", 2024-04-24T15:58:57.763 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_align_min_size": "65536", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_block_align": "true", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_block_size": "4096", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_dio": "true", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_discard": "false", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_force_aio": "false", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_ignore_corruption": "false", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_max_write_bytes": "10485760", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_max_write_entries": "100", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_replay_from": "0", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_write_header_frequency": "0", 2024-04-24T15:58:57.764 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_zero_on_create": "false", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_prefetch_periods": "10", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_prezero_periods": "5", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_write_head_interval": "15", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "keyfile": "", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_backend": "rocksdb", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_default_stripe_size": "65536", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_fsck_on_mount": "false", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_max_bytes": "67108864", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_max_ops": "512", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_nid_prealloc": "1024", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_onode_map_size": "1024", 2024-04-24T15:58:57.765 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_sync_submit_transaction": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_sync_transaction": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_block_size": "65536", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_bloom_size": "0", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_cache_size": "536870912", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_compact_on_mount": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_compression": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_log": "", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_log_to_ceph_log": "true", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_max_open_files": "0", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_paranoid": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "leveldb_write_buffer_size": "33554432", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "librados_thread_count": "2", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "lockdep": "false", 2024-04-24T15:58:57.766 INFO:teuthology.orchestra.run.smithi032.stdout: "lockdep_force_backtrace": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_coarse_timestamps": "true", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_flush_on_exit": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_graylog_host": "127.0.0.1", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_graylog_port": "12201", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_max_new": "1000", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_max_recent": "10000", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_stderr_prefix": "", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_stop_at_utilization": "0.970000", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": "true", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_graylog": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_journald": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_stderr": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_syslog": "false", 2024-04-24T15:58:57.767 INFO:teuthology.orchestra.run.smithi032.stdout: "max_rotating_auth_attempts": "10", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_action_on_write_error": "1", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_alternate_name_max": "8192", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_asio_thread_count": "2", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_export_pin": "true", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_dirs": "true", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_interval": "5", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_interval": "10", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_max": "-1", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_max_until": "-1", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_merge_size": "50", 2024-04-24T15:58:57.768 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_midchunk": "0.300000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_min_start": "0.200000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_minchunk": "0.001000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_mode": "0", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_need_max": "1.200000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_need_min": "0.800000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_bits": "3", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_size": "10000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_target_decay": "10.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_interval": "4.000000", 2024-04-24T15:58:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_mid": "0.700000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_release_free_interval": "10", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_reservation": "0.050000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_interval": "1", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_threshold": "262144", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_prealloc_inos": "1000", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-24T15:58:57.770 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_connect_bootstrapping": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_damage_table_max_entries": "10000", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_auth_pins": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_frag": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_scatterstat": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_subtrees": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_decay_halflife": "5.000000", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_default_dir_hash": "2", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_defer_session_stale": "true", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_deny_all_reconnect": "false", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_keys_per_op": "16384", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_max_commit_size": "10", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_max_entries": "0", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-24T15:58:57.771 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_on_map": "false", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_early_reply": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_enable_op_tracker": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_enforce_unique_name": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_random": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-24T15:58:57.772 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_health_summarize_threshold": "10", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_health_dummy": "false", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_migrator_session_race": "false", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_join_fs": "", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_journal_format": "1", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_create_at": "0", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_export_at": "0", 2024-04-24T15:58:57.773 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_import_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_expire_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_replay_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_link_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_mdstable_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_openc_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_rename_at": "0", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_events_per_segment": "1024", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_max_events": "-1", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_max_segments": "128", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_pause": "false", 2024-04-24T15:58:57.774 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_segment_size": "0", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_warn_factor": "2.000000", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_caps_per_client": "1048576", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_completed_flushes": "100000", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_completed_requests": "100000", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_export_size": "20971520", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_file_recover": "32", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_files": "64", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_ops": "8192", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_snaps_per_dir": "100", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_xattr_pairs_size": "65536", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_metrics_update_interval": "2", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_min_caps_per_client": "100", 2024-04-24T15:58:57.775 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_min_caps_working_set": "10000", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_numa_node": "-1", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_complaint_time": "30.000000", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_duration": "600", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_size": "20", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_slow_op_size": "20", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_log_threshold": "5", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_ping_grace": "15", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_ping_interval": "5", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_caps": "30000", 2024-04-24T15:58:57.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_warning_threshold": "262144", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_replay_interval": "1.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_root_ino_gid": "0", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_root_ino_uid": "0", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-24T15:58:57.777 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_shutdown_check": "0", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_skip_ino": "0", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_max_uid": "4294967294", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_min_uid": "0", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_rstat": "false", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_standby_replay_damaged": "false", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_symlink_recovery": "true", 2024-04-24T15:58:57.778 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_thrash_exports": "0", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_thrash_fragments": "0", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_tick_interval": "5.000000", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_valgrind_exit": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_verify_backtrace": "1", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_verify_scatter": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_wipe_sessions": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool_debug": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_device_bytes": "1073741824", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_page_set": "false", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_page_size": "65536", 2024-04-24T15:58:57.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_bytes": "134217728", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_messages": "512", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_disabled_modules": "", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_max_pg_num_change": "128", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mds_bytes": "134217728", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mds_messages": "128", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mon_bytes": "134217728", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mon_messages": "128", 2024-04-24T15:58:57.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_osd_bytes": "536870912", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_osd_messages": "8192", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_pool": "true", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_standby_modules": "true", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_stats_period": "5", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_stats_threshold": "5", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_tick_period": "2", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_allow_pool_delete": "true", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_allow_pool_size_one": "true", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_auth_validate_all_caps": "true", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-24T15:58:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_bytes": "104857600", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_directed_command_retry": "5", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_parallel": "3", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_log_interval": "1.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_ping_interval": "10.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-24T15:58:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_file_level": "debug", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_file": "true", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_journald": "false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_bootstrap": "false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_start": "false", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_trim": "true", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-24T15:58:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cpu_threads": "4", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_crush_min_required_version": "hammer", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_daemon_bytes": "419430400", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_avail_crit": "5", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_avail_warn": "5", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_size_warn": "16106127360", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_json": "false", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_transactions": "false", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_extra_checks": "false", 2024-04-24T15:58:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_pacific": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_quincy": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_down_mkfs_grace": "120", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_election_default_strategy": "1", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_election_timeout": "5.000000", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ping_divisor": "2", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_enable_op_tracker": "true", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_fake_pool_delete": "false", 2024-04-24T15:58:57.785 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_force_quorum_join": "false", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_globalid_prealloc": "10000", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_detail_to_clog": "false", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_log_update_period": "5", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_max_detail": "50", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog": "true", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog_interval": "600", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_host": "172.21.15.32,[v2:172.21.15.32:3301,v1:172.21.15.32:6790],172.21.15.86", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_host_override": "", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_initial_members": "", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-24T15:58:57.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease": "5.000000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_full_interval": "50", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_max": "10000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_max_summary": "50", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_log_epochs": "500", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_osd": "10000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_pg_per_osd": "10000", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_pool_pg_num": "65536", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-24T15:58:57.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_force_trim_to": "0", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_skip_sanity": "false", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_memory_autotune": "true", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_memory_target": "2147483648", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_beacon_grace": "30", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_digest_period": "5", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_inactive_grace": "60", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_min_osdmap_epochs": "500", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_complaint_time": "30", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_duration": "600", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_size": "20", 2024-04-24T15:58:57.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_slow_op_size": "20", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_log_threshold": "5", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_in": "false", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_cache_size": "500", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_down_out_interval": "600", 2024-04-24T15:58:57.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_force_trim_to": "0", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_down_reporters": "2", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-24T15:58:57.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_report_timeout": "900", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_stuck_threshold": "60", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-24T15:58:57.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_probe_timeout": "2.000000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_max_change": "0.050000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_max_osds": "4", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_interval": "86400", 2024-04-24T15:58:57.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_max_keys": "100", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_timeout": "300", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_session_timeout": "300", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_smart_report_timeout": "5", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stat_smooth_intervals": "6", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_pool_min_size": "2", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_pool_size": "4", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_debug": "false", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_provider_kill_at": "0", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_requester_kill_at": "0", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_timeout": "60.000000", 2024-04-24T15:58:57.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_target_pg_per_osd": "100", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_tick_interval": "5", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_timecheck_interval": "300.000000", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_older_version_delay": "604800", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_misplaced": "false", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-24T15:58:57.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_older_version": "true", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "monmap": "", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_op_threads": "3", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_cm": "false", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_device_name": "", 2024-04-24T15:58:57.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_dscp": "96", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_local_gid": "", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_port_num": "1", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_sl": "3", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_support_srq": "true", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_type": "ib", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_reap_threshold": "5", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_before_connect": "false", 2024-04-24T15:58:57.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_ipv4": "true", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_ipv6": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_msgr1": "true", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_msgr2": "true", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_port_max": "7568", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_port_min": "6800", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_retry_count": "3", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_retry_delay": "5", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_client": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mds": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mgr": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mon": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_osd": "false", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_client_mode": "crc secure", 2024-04-24T15:58:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_cluster_mode": "crc secure", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_cluster_type": "", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_compress_secure": "false", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_connection_idle_timeout": "900", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_connection_ready_timeout": "10", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_crc_data": "true", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_crc_header": "true", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_bad_msg": "false", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_bug": "true", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_old_message": "true", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_skipped_message": "false", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_coremask": "0xF", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-24T15:58:57.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_enable_tso": "true", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hugepages": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_lro": "true", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_memory_channel": "4", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_pmd": "", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_port_id": "0", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dump_on_send": "false", 2024-04-24T15:58:57.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_initial_backoff": "0.200000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_max": "1.000000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_type": "", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_socket_failures": "5000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_learn_addr_from_peer": "true", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_max_accept_failures": "4", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_max_backoff": "15.000000", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_client_mode": "secure crc", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_service_mode": "secure crc", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compress_min_size": "1024", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compress_mode": "none", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-04-24T15:58:57.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_pq_min_cost": "65536", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_public_type": "", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_service_mode": "crc secure", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_listen_backlog": "512", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_nodelay": "true", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_rcvbuf": "0", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_type": "async+posix", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "no_config_file": "false", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_completion_locks_per_session": "32", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inflight_ops": "1024", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-24T15:58:57.801 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_tick_interval": "5.000000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_timeout": "10.000000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "objectstore_blackhole": "false", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "openssl_engine_opts": "", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_delay_time": "5.000000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_hist_halflife": "1000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_max_low_ops": "2", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_max_ops": "4", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_slop": "0.020000", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_asio_thread_count": "2", 2024-04-24T15:58:57.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_async_recovery_min_cost": "100", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_scan_max": "512", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_scan_min": "64", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_degraded": "false", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_peering": "false", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_unfound": "true", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_beacon_report_interval": "300", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_duration": "30", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_max_block_size": "67108864", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_blkin_trace_all": "false", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_check_for_log_corruption": "false", 2024-04-24T15:58:57.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_update_on_start": "true", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_message_cap": "256", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_message_size_cap": "524288000", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_op_priority": "63", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_watch_timeout": "30", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_command_max_records": "256", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_compact_on_start": "false", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_update_on_start": "true", 2024-04-24T15:58:57.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_update_weight_set": "true", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_feed_pullee": "-1", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_misdirected_ops": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_no_acting_change": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_no_purge_strays": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_op_order": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-24T15:58:57.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_shutdown": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_snaps": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_keys": "1024", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-24T15:58:57.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_stride": "524288", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_default_notify_timeout": "30", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep": "0.000000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_discard_disconnected_ops": "true", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_enable_op_tracker": "true", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_info": "true", 2024-04-24T15:58:57.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown": "true", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_function_tracing": "false", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_grace": "20", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_interval": "6", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_peers": "10", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_size": "2000", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_stale": "600", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-24T15:58:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_max_size": "100000", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_min_size": "1000", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal_size": "5120", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_kill_backfill_at": "0", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_cache_size": "50", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_dedup": "true", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_message_max": "40", 2024-04-24T15:58:57.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_share_max_epochs": "40", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_attr_name_len": "100", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_attr_size": "0", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_backfills": "1", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_markdown_count": "5", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_markdown_period": "600", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_name_len": "2048", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_namespace_len": "256", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_size": "134217728", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_blocked_by": "16", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_log_entries": "10000", 2024-04-24T15:58:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pgls": "1024", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_push_cost": "8388608", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_push_objects": "10", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_scrubs": "3", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_trimming_pgs": "2", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_write_op_reply_len": "64", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_write_size": "90", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-24T15:58:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_profile": "balanced", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_base": "805306368", 2024-04-24T15:58:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_cache_min": "134217728", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target": "4294967296", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target_autotune": "false", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_min_pg_log_entries": "250", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_min_recovery_priority": "0", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_report_interval": "5", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_num_cache_shards": "32", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_num_op_tracker_shard": "32", 2024-04-24T15:58:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_auto_affinity": "true", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_node": "-1", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_prefer_iface": "true", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objecter_finishers": "1", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore": "bluestore", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore_fuse": "false", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore_tracing": "false", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_complaint_time": "30.000000", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_duration": "600", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_size": "20", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_slow_op_size": "20", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_log_threshold": "5", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards": "0", 2024-04-24T15:58:57.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards_hdd": "5", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards_ssd": "8", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_pq_min_cost": "65536", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_queue_cut_off": "high", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_thread_timeout": "15", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_open_classes_on_start": "true", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_os_flags": "0", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_peering_op_priority": "255", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_delete_cost": "1048576", 2024-04-24T15:58:57.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_delete_priority": "5", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_trim_max": "10000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_trim_min": "100", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_stat_report_interval_max": "500", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-24T15:58:57.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flags": "0", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_min_size": "0", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pg_num": "32", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pgp_num": "0", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-24T15:58:57.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_size": "2", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_type": "replicated", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_push_per_object_cost": "1000", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recover_clone_overlap": "true", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_cost": "20971520", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active": "0", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-24T15:58:57.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_single_start": "1", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_op_priority": "3", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_priority": "5", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep": "0.000000", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_repair_during_recovery": "false", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_requested_scrub_priority": "120", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_auto_repair": "false", 2024-04-24T15:58:57.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_begin_hour": "0", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_begin_week_day": "0", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_chunk_max": "25", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_chunk_min": "5", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_cost": "52428800", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_during_recovery": "false", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_end_hour": "0", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_end_week_day": "0", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_event_cost": "4096", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_invalid_stats": "true", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-24T15:58:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_max_preemptions": "5", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_priority": "5", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_sleep": "0.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_skip_data_digest": "false", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_smart_report_timeout": "5", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_cost": "1048576", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_priority": "5", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_target_transaction_size": "30", 2024-04-24T15:58:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tracing": "false", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_use_stale_snap": "false", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osdc_blkin_trace_all": "false", 2024-04-24T15:58:57.822 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_kill_at": "0", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_max_join_drift": "10", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_min": "500", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_min_wait": "0.050000", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_propose_interval": "1.000000", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_max": "500", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_min": "250", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_stash_full_interval": "25", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_trim_max": "500", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_trim_min": "250", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "perf": "true", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-24T15:58:57.823 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "-", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrv": "", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "public_bind_addr": "-", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "public_network": "", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "public_network_interface": "", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "qat_compressor_enabled": "false", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_mon_op_timeout": "0", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_osd_op_timeout": "0", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_tracing": "false", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_atime_update_interval": "60", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_balance_parent_reads": "false", 2024-04-24T15:58:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_balance_snap_reads": "false", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blkin_trace_all": "false", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache": "true", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-24T15:58:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_policy": "writearound", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_size": "33554432", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_clone_copy_on_read": "false", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_compression_hint": "none", 2024-04-24T15:58:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_concurrent_management_ops": "10", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_clone_format": "auto", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_data_pool": "", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_features": "61", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_format": "2", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_map_options": "", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_order": "22", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_pool": "rbd", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-24T15:58:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_stripe_count": "0", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_stripe_unit": "0", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_enable_alloc_hint": "true", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_io_scheduler": "simple", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-24T15:58:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_order": "24", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_pool": "", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_splay_width": "4", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_localize_parent_reads": "false", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_localize_snap_reads": "false", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-24T15:58:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-24T15:58:57.830 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mtime_update_interval": "60", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_non_blocking_aio": "true", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_op_thread_timeout": "60", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_op_threads": "1", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_parent_cache_enabled": "false", 2024-04-24T15:58:57.831 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_plugins": "", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_burst": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_limit": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_exclude_ops": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_burst": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_limit": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-24T15:58:57.832 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_read_from_replica_policy": "default", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_skip_partial_discard": "true", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-24T15:58:57.833 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_tracing": "false", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_validate_names": "true", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_validate_pool": "true", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "restapi_base_url": "", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "restapi_log_level": "", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_acl_grants_max_num": "100", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_admin_entry": "admin", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_backend_store": "rados", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_barbican_url": "", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_beast_enable_async": "true", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-24T15:58:57.834 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_enabled": "true", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_expiry_interval": "900", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_lru_size": "10000", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_content_length_compat": "false", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_copy_obj_progress": "true", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cors_rules_max_num": "100", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cross_domain_policy": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_addr": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-24T15:58:57.835 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_password": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_username": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_require_ssl": "true", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-24T15:58:57.836 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_addr": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_auth": "token", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_namespace": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_prefix": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_token_file": "", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_buffersize": "524288", 2024-04-24T15:58:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_low_speed_time": "300", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_changes_size": "1000", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_num_shards": "128", 2024-04-24T15:58:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_window": "30", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_notify_interval_msec": "200", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-24T15:58:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dns_name": "", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dns_s3website_name": "", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dynamic_resharding": "true", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_gc_threads": "true", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_lc_threads": "true", 2024-04-24T15:58:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_ops_log": "false", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_quota_threads": "true", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_static_website": "false", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_usage_log": "false", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enforce_swift_acls": "true", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_exit_timeout_secs": "120", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_expose_bucket": "false", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_extended_http_attrs": "", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_frontends": "beast port=7480", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_deferred": "50", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_objs": "32", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-24T15:58:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_processor_period": "3600", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_init_timeout": "300", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_domain": "", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_password": "", 2024-04-24T15:58:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_password_path": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_project": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_tenant": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_token": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_token_path": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_user": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_api_version": "2", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_domain": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_password": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_project": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_user": "", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-24T15:58:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_url": "", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_debug_interval": "-1", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_lock_max_time": "90", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_objs": "32", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_rules": "1000", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_worker": "3", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_thread_delay": "0", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_dnattr": "uid", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_searchfilter": "", 2024-04-24T15:58:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_uri": "", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_http_headers": "", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_object_name_utc": "false", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attr_name_len": "0", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attr_size": "0", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_chunk_size": "4194304", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-24T15:58:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_listing_results": "1000", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_notify_retries": "10", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_put_param_size": "1048576", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_put_size": "5368709120", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_slo_entries": "1000", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_md_log_max_shards": "64", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_mp_lock_max_time": "600", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-24T15:58:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_max_gc": "300", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_num_async_rados_threads": "32", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_num_control_oids": "8", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_numa_node": "-1", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-24T15:58:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_chunk_size": "100", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_gc_interval": "600", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_thread_timeout": "600", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_tracing": "false", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_token": "", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_url": "", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_verify_ssl": "true", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_file_path": "", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_rados": "true", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_socket_path": "", 2024-04-24T15:58:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_push_interval": "2.000000", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_print_continue": "true", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_tracing": "false", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm": "", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm_id": "", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-24T15:58:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_region": "", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_request_uri": "", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_batch_size": "64", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_max_aio": "128", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_num_logs": "16", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_thread_interval": "600", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_resolve_cname": "false", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_run_sync_thread": "true", 2024-04-24T15:58:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_scheduler_type": "throttler", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_script_uri": "", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_service_provider_name": "", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_client_id": "", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_client_secret": "", 2024-04-24T15:58:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_entry": "sts", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_key": "sts", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_min_session_duration": "900", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_token_introspection_url": "", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_account_in_url": "false", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_auth_entry": "auth", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_auth_url": "", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_custom_header": "", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_need_stats": "true", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_tenant_name": "", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_token_expiration": "86400", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_url": "", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_url_prefix": "swift", 2024-04-24T15:58:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_lease_period": "120", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_thread_pool_size": "512", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_comment": "", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_createby": "", 2024-04-24T15:58:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_encoding": "", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_flag": "false", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_origin": "", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_tracker": "", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_trust_forwarded_https": "false", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_max_shards": "32", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_max_user_shards": "1", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_use_opa_authz": "false", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_max_buckets": "1000", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-24T15:58:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_unique_email": "true", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_verify_ssl": "true", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone": "", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone_id": "", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup": "", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup_id": "", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_block_size": "4096", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-24T15:58:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_size": "536870912", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_index_type": "binary_search", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-24T15:58:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_partition_filters": "false", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_perf": "false", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "run_dir": "/var/run/ceph", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_block_create": "true", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_cache_lru_size": "67108864", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_default_max_object_size": "16777216", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_device_size": "107374182400", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_capacity": "16", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-24T15:58:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_segment_size": "67108864", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "setgroup": "", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "setuser": "", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "setuser_match_path": "", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "thp": "false", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "threadpool_default_timeout": "60", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "throttler_perf_counter": "true" 2024-04-24T15:58:57.858 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:58:57.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-24T15:58:58.013 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a: {} 2024-04-24T15:58:58.024 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b: {} 2024-04-24T15:58:58.034 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c: {} 2024-04-24T15:58:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T15:58:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T15:58:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T15:58:58.048 DEBUG:teuthology.orchestra.run.smithi086:> type ceph-objectstore-tool 2024-04-24T15:58:58.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T15:58:58.069 DEBUG:teuthology.orchestra.run.smithi032:> type ceph-objectstore-tool 2024-04-24T15:58:58.089 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T15:58:58.089 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T15:58:58.102 INFO:tasks.workunit:Pulling workunits from ref 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:58:58.102 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T15:58:58.102 DEBUG:teuthology.orchestra.run.smithi122:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T15:58:58.103 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T15:58:58.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T15:58:58.104 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-24T15:58:58.104 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-24T15:58:58.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:58:58.105 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-24T15:58:58.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:58:58.105 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-24T15:58:58.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:58:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:58:58.126 INFO:teuthology.orchestra.run.smithi122.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T15:58:58.126 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-24T15:58:58.131 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:58.132+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:58:58.187 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-24T15:58:58.188 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-24T15:58:58.206 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:58.208+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:58:58.238 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:58:58.238 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T15:58:58.238 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:58:58.247 INFO:tasks.workunit:timeout=3h 2024-04-24T15:58:58.248 INFO:tasks.workunit:cleanup=True 2024-04-24T15:58:58.248 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:58:58.283 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:58:58.283 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:58:58.283 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:58:58.283 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:58:58.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:58:58.306 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:58:58.307 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:58.309+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:58:58.309 INFO:tasks.workunit.client.0.smithi122.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T15:58:58.326 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:58:58.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:58:58.367 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:58:58.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:58:58.406 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:58:58.407 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:58.409+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:58:58.447 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:58.447 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:58.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:58:58.507 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:58.509+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:58:58.609 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:58.610+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:58:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:58.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:58:58.734 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:58.736+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:58:58.811 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:58.813+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:58:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:58.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:58:58.927 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:58.929+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:58:59.012 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:59.014+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:58:59.025 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.025 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:58:59.094 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:58:59.113 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:59.115+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:58:59.214 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:58:59.216+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:58:59.221 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.222 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:58:59.315 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:58:59.317+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:58:59.373 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.373 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:58:59.416 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:58:59.417+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:58:59.516 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:59.518+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:58:59.566 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.566 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:58:59.618 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:59.620+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:58:59.718 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:58:59.720+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:58:59.759 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.760 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:58:59.819 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:58:59.822+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:58:59.910 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:58:59.959 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:58:59.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:58:59.961 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:58:59.962+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:00.061 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:00.064+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:00.125 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:00.126 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:00.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:00.162 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:00.165+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:00.258 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:00.258 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:00.263 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:00.265+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:00.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:00.364 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:00.366+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:00.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:00.431 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:00.431 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:00.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:00.466 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:00.468+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:00.566 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:00.568+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:00.586 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:00.605 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:00.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:00.624 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:00.645 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:00.665 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:00.667 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:00.669+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:00.684 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:00.686 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:00.740 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:00.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:00.740 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:00.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:00.794 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:00.796+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:00.869 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:00.871+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:00.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:00.971 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:00.973+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:01.049 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:01.049 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:01.049 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:01.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:01.072 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:01.074+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:01.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T15:59:01.172 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:01.174+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:01.183 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:01.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:01.273 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:01.275+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:01.337 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:01.337 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:01.374 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:01.375+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:01.475 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:01.477+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:01.526 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:01.527 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:01.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:01.575 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:01.578+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:01.677 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:01.679+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:01.719 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:01.719 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:01.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:01.778 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:01.780+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:01.870 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:01.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:01.871 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:01.871 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:01.879 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:01.881+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:01.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:01.980 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:01.982+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:02.063 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:02.064 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:02.064 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:02.064 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:02.064 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:02.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:02.081 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:02.083+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T15:59:02.182 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:02.185+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:02.219 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.188374112000002, 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:02.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:02.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:02.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:02.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.167128025, 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:02.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:02.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:02.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.152945763000002, 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:02.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:02.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:02.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.231 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.139447987, 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:02.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.138938677999999, 2024-04-24T15:59:02.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:02.240 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.125609396000002, 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.125103677999999, 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:02.248 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.103546550000001, 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.957278981000002, 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:02.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.956025150999999, 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.257 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:02.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.952919973, 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:02.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.265 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.952482595999999, 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.952042844000001, 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.635242782999999, 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.634703570999999, 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.631066577999999, 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:02.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.629871202, 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.629460905999998, 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.189721479999999, 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:02.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.187116656000001, 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:02.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:02.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:02.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:02.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:02.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:02.302 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:02.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:02.304 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:02.305+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:02.404 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:02.406+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:02.490 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:02.490 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:02.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:02.505 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:02.507+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:02.606 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:02.608+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:02.639 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:02.639 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:02.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:02.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:02.707 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:02.709+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:02.796 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:02.796 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:02.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:02.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:02.797 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:02.807 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:02.809+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:02.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:02.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:02.877 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:02.896 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:02.908 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:02.910+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:02.916 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:02.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:02.956 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:02.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:03.043 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:03.045+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:03.110 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:03.112+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:03.211 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:03.214+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:03.250 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:03.250 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:03.250 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-24T15:59:03.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-24T15:59:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:03.351 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:03.353+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:03.413 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:03.415+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-24T15:59:03.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:03.516 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:03.518+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:03.615 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:03.616+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:03.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-24T15:59:03.716 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:03.717+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:03.816 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:03.818+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:03.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-24T15:59:03.917 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:03.919+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:03.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:04.048 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:04.050+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:04.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-24T15:59:04.119 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:04.120+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:04.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T15:59:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:04.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:04.220 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:04.222+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:04.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-24T15:59:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:04.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:04.344 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:04.346+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:04.422 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:04.424+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:04.486 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:04.486 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:04.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:04.539 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:04.541+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:04.624 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:04.626+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:04.681 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:04.682 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:04.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:04.734 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:04.736+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:04.825 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:04.828+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:04.874 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:04.874 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:04.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:04.927 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:04.928+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:04.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:05.027 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.029+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:05.028 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:05.028 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:05.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T15:59:05.129 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:05.131+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:05.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:05.165 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:05.184 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:05.205 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:05.224 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:05.224 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:05.225 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:05.225 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:05.225 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:05.225 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:05.229 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.231+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:05.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:05.245 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:05.330 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:05.332+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.354651788999998, 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:05.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:05.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:05.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.333405702, 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:05.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.319223440000002, 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.305725664000001, 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:05.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.305216354999999, 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.404 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:05.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:05.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.291887073000002, 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.291381354999999, 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:05.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:05.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.269824227000001, 2024-04-24T15:59:05.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:05.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.123556657999998, 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:05.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:05.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:05.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.122302827999999, 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.430 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:05.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:05.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:05.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.11919765, 2024-04-24T15:59:05.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:05.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:05.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.118760272999999, 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:05.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.118320521000001, 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.801520459999999, 2024-04-24T15:59:05.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.800981247999999, 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:05.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.797344254999999, 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:05.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:05.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.796148879, 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.795738582999999, 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.355999156999999, 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.353394333000001, 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:05.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:05.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:05.479 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:05.479 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:05.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:05.480 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:05.482+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:05.580 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.582+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:05.667 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:05.667 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:05.667 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:05.667 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:05.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:05.681 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:05.683+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:05.782 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.784+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:05.861 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:05.861 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:05.861 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:05.861 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:05.861 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:05.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:05.915 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.917+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:05.983 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:05.985+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:06.056 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:06.056 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:06.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:06.108 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:06.110+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:06.185 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:06.187+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:06.207 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:06.207 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:06.207 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:06.207 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:06.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:06.285 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:06.288+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:06.340 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:06.340 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:06.340 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:06.340 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:06.340 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:06.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:06.386 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:06.388+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:06.474 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:06.474 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:06.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.380359582000001, 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.475 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:06.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:06.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.480 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.373158417999999, 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:06.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:06.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.373101032000001, 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.216016707000001, 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.492 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:06.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:06.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:06.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215937357000001, 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.498 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215631886000001, 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215610743999999, 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:06.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215516847, 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215504783, 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215487358000001, 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215421588000002, 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.215411119999999, 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889889798999999, 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889792642, 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:06.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889770399, 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889736471999999, 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889620247, 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889593426000001, 2024-04-24T15:59:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889571367999999, 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.889505623000002, 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:06.589 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:06.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:06.591 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:06.592+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:06.691 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:06.694+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:06.754 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:06.754 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:06.754 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:06.754 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:06.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:06.807 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:06.809+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:06.887 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:06.888 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:06.888 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:06.892 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:06.893+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:06.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:06.993 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:06.995+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:07.061 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:07.061 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:07.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:07.062 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:07.062 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:07.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:07.112 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:07.115+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:07.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T15:59:07.195 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:07.197+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:07.234 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:07.235 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:07.235 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:07.235 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:07.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:07.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:07.304 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:07.306+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:07.368 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:07.368 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:07.368 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:07.368 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:07.368 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:07.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:07.397 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:07.399+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:07.416 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:07.437 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:07.456 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:07.476 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:07.495 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:07.498 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:07.501+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:07.503 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:07.503 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:07.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:07.503 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:07.503 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:07.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:07.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:07.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:07.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:07.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:07.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:07.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:07.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:07.599 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:07.601+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:07.652 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:07.652 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:07.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:07.700 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:07.703+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:07.801 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:07.803+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:07.844 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:07.844 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:07.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:07.902 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:07.904+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:08.003 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:08.006+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:08.037 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.037 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:08.104 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:08.106+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T15:59:08.191 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.191 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:08.205 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:08.207+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:08.306 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:08.308+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:08.341 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.341 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:08.438 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:08.440+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:08.508 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:08.510+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:08.536 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.536 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:08.608 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:08.611+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:08.709 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:08.711+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:08.729 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.730 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:08.825 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:08.827+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:08.911 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:08.913+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:08.923 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:08.923 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:08.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:09.012 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:09.014+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:09.112 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:09.114+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:09.115 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:09.115 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:09.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:09.213 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:09.215+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:09.263 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:09.264 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:09.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:09.314 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:09.317+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:09.397 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:09.397 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:09.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:09.451 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:09.454+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:09.476 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:09.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:09.476 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-24T15:59:09.476 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T15:59:09.476 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T15:59:09.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-24T15:59:09.516 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:09.518+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:09.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:09.534 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:09.534 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:09.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:09.606 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:09.606 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T15:59:09.606 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:09.617 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:09.618+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:09.669 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:09.670 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:09.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:09.695 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:09.714 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:09.723 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:09.726+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:09.733 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:09.754 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:09.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:09.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:09.794 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:09.804 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:09.805 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:09.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:09.848 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:09.850+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:09.919 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:09.922+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:09.979 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:09.979 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:09.979 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:09.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:10.029 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:10.032+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:10.121 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:10.124+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:10.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:59:10.153 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:10.154 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:10.154 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:10.154 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:10.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:10.222 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:10.224+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:10.323 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:10.326+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:10.327 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:10.328 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:10.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:10.426 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:10.428+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:10.500 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:10.501 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:10.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:10.501 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:10.501 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:10.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:10.551 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:10.554+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:10.627 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:10.630+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:10.648 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:10.648 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:10.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:10.728 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:10.730+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:10.797 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:10.797 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:10.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:10.829 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:10.831+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:10.930 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:10.932+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:10.988 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:10.988 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:10.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:10.988 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:10.988 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:10.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:11.042 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:11.043+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:11.131 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:11.133+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:11.132 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:59:11.185 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:11.185 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:11.185 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:11.185 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:11.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:11.238 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:11.240+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:11.333 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:11.335+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:11.379 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:11.379 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:11.379 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:11.379 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:11.379 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:11.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:11.433 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:11.436+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:11.534 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:11.537+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.542670743999999, 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.574 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:11.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.521424657000001, 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.578 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:11.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:11.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:11.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.507242394999999, 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:11.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:11.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:11.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.493744619000001, 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.586 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:11.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.493235309999999, 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:11.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:11.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.479906027999998, 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:11.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:11.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:11.601 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.479400309999999, 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:11.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:11.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:11.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.606 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.457843182000001, 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:11.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:11.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.311575612999999, 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:11.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:11.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.310321782999999, 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:11.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.307216605000001, 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:11.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:11.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:11.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.306779228, 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:11.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.306339476000002, 2024-04-24T15:59:11.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:11.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.989539414999999, 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.989000203, 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.985363209999999, 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:11.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:11.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:11.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.984167834000001, 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.983757537999999, 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:11.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:11.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.544018112, 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:11.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:11.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.541413288000001, 2024-04-24T15:59:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:11.662 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:11.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:11.665 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:11.666+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:11.765 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:11.767+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:11.807 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:11.807 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:11.807 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:11.807 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:11.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:11.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:11.866 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:11.869+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:11.964 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:11.967 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:11.969+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:11.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:12.000 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:12.001 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:12.001 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:12.001 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:12.001 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:12.004 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:12.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:12.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:12.045 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:12.066 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:12.069 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:12.071+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:12.165 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.165 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.169 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:12.172+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:12.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:12.270 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:12.272+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:12.313 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.313 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:12.371 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:12.374+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:12.446 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.447 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:12.499 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:12.501+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:12.575 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:12.578+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:12.620 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.620 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:12.677 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:12.679+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:12.753 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.753 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:12.778 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:12.781+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:12.879 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:12.882+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:12.885 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:12.886 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:12.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:12.980 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:12.982+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:13.018 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.019 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:13.081 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:13.084+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:13.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T15:59:13.182 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:13.185+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:13.195 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.195 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:13.283 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:13.285+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:13.368 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.368 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:13.384 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:13.386+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:13.485 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:13.487+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:13.500 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.500 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:13.585 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:13.588+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:13.650 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.650 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:13.705 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:13.707+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:13.787 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:13.789+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.799 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:13.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:13.894 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:13.896+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:13.988 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:13.991+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:13.991 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:13.991 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:14.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:14.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:14.089 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:14.092+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:14.146 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T15:59:14.190 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:14.193+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:14.197 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:14.197 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:14.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:14.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:14.268 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:14.287 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:14.291 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:14.293+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:14.308 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:14.327 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:14.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:14.349 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:14.387 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:14.387 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:14.387 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:14.387 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:14.387 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:14.391 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:14.393+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:14.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:14.493 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:14.496+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.551715844, 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.530469756999999, 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.516287495, 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.502789718999999, 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.502280410000001, 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.488951128, 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.488445410000001, 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.466888281999999, 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.320620713, 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.319366883000001, 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.316261704999999, 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.315824328000001, 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.315384576, 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.998584515000001, 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.998045303000001, 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.994408310000001, 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.993212933999999, 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.992802638000001, 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:14.654 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.553063212000001, 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:14.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.550458387999999, 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:14.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:14.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:14.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:14.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:14.662 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:14.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:14.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:14.663 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T15:59:14.663 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T15:59:14.663 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T15:59:14.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T15:59:14.665 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:14.666+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:14.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:14.764 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:14.767+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:14.791 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:14.792 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T15:59:14.792 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:14.850 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:14.851 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:14.851 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:14.851 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:14.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:14.865 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:14.867+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:14.965 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:14.967+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:15.001 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:15.001 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:15.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:15.096 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:15.099+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:15.167 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:15.169+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:15.194 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:15.194 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:15.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:15.268 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:15.271+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:15.336 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:15.336 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:15.336 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:15.336 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:15.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:15.385 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:15.388+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:15.463 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:15.464 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:15.464 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:15.464 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:15.464 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:15.470 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:15.473+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:15.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:15.571 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:15.574+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.544866497000001, 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:15.639 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:15.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:15.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:15.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.537665333, 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.645 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:15.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:15.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:15.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.537607946999998, 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:15.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:15.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:15.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380523621999998, 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.660 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380444271999998, 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380138801000001, 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380117659, 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380023762, 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:15.681 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380011697999997, 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:15.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.686 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.379994273000001, 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.379928503000002, 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:15.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:15.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:15.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:15.697 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.379918035000003, 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054396713999999, 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:15.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:15.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054299557, 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054277313999997, 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.714 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:15.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:15.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:15.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054243387, 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054127162, 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.725 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:15.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:15.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:15.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054100341000002, 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:15.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:15.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:15.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:15.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054078283000003, 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:15.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:15.741 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.054012538000002, 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:15.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:15.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:15.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:15.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:15.747 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:15.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:15.749 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:15.751+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:15.849 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:15.851+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:15.911 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:15.912 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:15.912 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:15.912 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:15.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:15.950 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:15.953+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:16.051 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:16.054+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:16.084 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:16.084 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:16.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:16.153 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:16.156+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:16.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T15:59:16.217 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:16.217 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:16.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:16.217 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:16.217 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:16.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:16.270 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:16.273+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:16.350 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:16.351 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:16.354 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:16.357+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:16.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:16.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:16.454 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:16.457+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:16.515 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:16.524 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:16.524 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:16.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:16.535 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:16.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:16.555 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:16.558+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:16.575 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:16.594 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:16.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:16.659 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:16.659 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:16.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:16.670 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:16.672+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:16.757 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:16.760+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:16.810 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:16.810 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:16.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:16.858 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:16.860+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:16.959 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:16.961+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:17.002 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.003 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:17.059 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:17.062+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T15:59:17.155 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.155 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.160 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:17.163+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:17.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:17.262 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:17.265+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:17.349 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.350 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:17.403 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:17.406+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:17.464 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:17.467+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:17.544 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.544 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:17.599 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:17.602+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:17.666 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:17.669+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:17.695 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.695 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:17.791 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:17.794+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:17.868 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:17.871+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:17.890 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:17.890 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:17.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:17.986 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:17.989+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:18.070 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:18.073+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:18.083 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.083 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:18.178 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:18.181+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:18.272 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:18.275+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:18.276 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.276 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:18.374 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:18.377+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:18.423 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.424 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:18.477 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:18.480+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:18.556 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.557 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:18.576 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:18.579+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:18.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:18.677 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:18.680+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:18.732 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.732 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:18.785 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:18.788+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:18.794 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:18.813 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:18.833 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:18.853 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:18.866 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:18.866 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:18.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:18.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:18.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:18.919 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:18.921+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:18.981 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:18.984+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:19.040 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:19.041 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:19.041 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:19.041 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:19.041 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:19.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:19.081 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:19.084+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:19.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T15:59:19.175 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:19.175 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:19.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:19.176 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:19.176 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:19.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:19.210 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:19.213+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:19.282 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:19.285+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:19.309 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:19.310 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:19.310 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:19.310 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:19.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:19.383 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:19.386+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:19.442 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:19.442 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:19.442 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:19.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:19.484 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:19.487+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:19.574 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:19.575 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:19.585 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:19.587+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:19.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:19.685 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:19.688+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:19.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:19.786 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:19.789+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:19.802 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:19.802 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:19.802 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T15:59:19.802 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-04-24T15:59:19.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T15:59:19.875 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:19.875 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:19.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:19.888 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:19.889+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:19.988 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:19.991+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:20.028 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.028 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:20.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:20.029 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:20.029 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:20.090 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:20.092+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:20.160 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T15:59:20.188 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:20.188 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.189 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:20.192+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:20.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:20.291 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:20.294+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:20.381 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:20.382 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.392 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:20.395+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:20.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:20.493 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:20.495+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:20.576 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:20.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.545266943999998, 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.524020856999996, 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.509838594999998, 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:20.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.496340818999997, 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.495831510000002, 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.482502228000001, 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.601 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.481996510000002, 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.460439381999997, 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.314171813000002, 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.312917982999998, 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.309812805, 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.618 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.309375428000003, 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.308935675999997, 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:20.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:20.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:20.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.992135615000002, 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:20.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:20.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:20.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:20.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.991596403000003, 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:20.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:20.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.987959410000002, 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.986764033999997, 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:20.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.986353737999998, 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:20.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:20.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.546614312000003, 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.544009488, 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:20.659 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.660 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:20.663+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:20.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:20.761 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:20.764+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:20.804 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:20.804 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:20.804 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:20.862 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:20.864+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:20.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:20.955 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:20.955 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:20.956 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:20.963 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:20.966+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:20.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:21.064 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:21.066+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:21.084 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:21.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:21.138 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:21.160 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T15:59:21.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:21.165 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:21.167+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:21.178 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.178 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.189 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:21.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:21.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:21.211 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:21.211 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:21.265 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:21.268+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:21.322 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.323 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:21.373 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:21.376+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:21.467 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:21.470+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:21.495 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.495 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:21.569 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:21.571+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:21.629 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.629 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:21.680 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:21.682+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:21.770 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:21.773+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:21.802 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.802 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:21.872 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:21.874+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:21.936 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:21.936 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:21.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:21.973 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:21.976+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:22.074 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:22.077+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:22.109 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.110 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:22.175 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:22.178+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:22.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:59:22.243 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.243 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:22.276 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:22.279+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:22.376 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.377 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.377 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:22.379+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:22.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:22.477 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:22.480+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:22.549 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.550 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:22.578 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:22.580+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:22.678 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:22.681+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:22.699 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.699 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:22.780 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:22.783+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:22.848 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:22.848 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:22.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:22.900 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:22.903+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:22.982 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:22.985+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:23.040 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:23.041 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:23.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:23.093 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:23.096+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:23.166 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:59:23.184 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:23.186+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:23.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:23.238 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:23.238 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:23.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:23.292 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:23.295+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:23.380 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:23.385 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:23.388+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:23.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:23.422 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:23.434 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:23.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:23.434 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:23.434 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:23.434 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:23.442 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:23.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:23.463 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:23.486 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:23.486 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:23.489+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:23.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:23.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:23.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:23.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:23.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:23.488 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:23.587 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:23.590+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:23.603 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:23.603 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:23.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.572422748000001, 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.551176661, 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.536994399000001, 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.523496623, 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.522987313999998, 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:23.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:23.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:23.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.509658031999997, 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:23.627 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.509152313999998, 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.631 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.487595186, 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:23.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.341327616999997, 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.340073787000001, 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:23.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.336968609000003, 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:23.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:23.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:23.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.647 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.336531231999999, 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:23.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:23.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.33609148, 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.019291418999998, 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:23.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:23.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.659 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.018752206999999, 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:23.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:23.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.663 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.015115213999998, 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:23.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:23.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:23.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.013919838, 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.671 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.013509542000001, 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.573770115999999, 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.571165292000003, 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:23.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:23.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:23.684 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:23.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:23.688 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:23.691+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:23.788 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:23.791+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:23.828 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:23.828 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:23.828 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:23.829 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:23.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:23.889 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:23.891+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:23.990 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:23.993+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:24.021 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:24.022 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:24.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:24.091 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:24.093+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:24.173 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:24.173 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:24.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:24.173 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:24.173 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:24.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:24.191 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:24.194+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:24.292 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:24.295+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:24.321 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:24.322 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:24.322 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:24.322 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:24.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:24.394 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:24.395+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:24.456 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:24.456 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:24.456 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:24.456 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:24.456 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:24.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:24.494 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:24.497+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:24.595 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:24.598+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.631 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.537370434000003, 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:24.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:24.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.643 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:24.644 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.530169270000002, 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.645 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:24.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:24.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:24.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:24.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:24.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.530111884, 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:24.662 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.373027559000001, 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372948209, 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:24.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372642738000003, 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.673 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372621596000002, 2024-04-24T15:59:24.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:24.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372527699000003, 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:24.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372515634999999, 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:24.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372498210000003, 2024-04-24T15:59:24.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372432439999997, 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.372421971999998, 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.705 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:24.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:24.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046900651000001, 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.710 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046803494000002, 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:24.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046781250999999, 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.721 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:24.726 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046747324000002, 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046631099000003, 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.732 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:24.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046604277999997, 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046582219999998, 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.046516474999997, 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:24.748 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:24.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:24.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:24.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:24.753 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:24.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:24.756 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:24.758+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:24.855 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:24.858+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:24.919 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:24.919 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:24.919 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:24.919 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:24.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:24.972 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:24.975+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:25.051 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:25.052 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:25.052 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:25.052 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:25.052 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:25.057 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:25.059+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:25.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:25.158 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:25.161+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:25.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T15:59:25.226 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:25.226 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:25.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:25.226 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:25.226 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:25.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:25.279 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:25.282+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:25.360 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:25.360 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:25.360 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:25.360 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:25.361 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:25.363+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:25.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:25.461 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:25.464+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:25.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:25.534 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:25.535 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:25.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:25.585 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:25.588+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:25.662 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:25.665+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:25.709 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:25.709 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:25.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:25.709 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:25.709 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:25.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:25.756 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:25.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:25.763 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:25.766+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:25.860 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:25.860 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:25.864 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:25.867+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:25.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:25.965 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:25.968+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:26.009 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.009 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:26.066 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:26.069+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:26.167 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:26.170+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:26.178 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:26.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:26.178 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.3082109227577033 2024-04-24T15:59:26.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 0.3082109227577033 2024-04-24T15:59:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T15:59:26.210 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.210 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:26.268 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:26.270+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:26.368 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:26.371+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:26.405 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.405 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:26.501 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:26.504+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:26.570 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:26.573+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:26.597 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.598 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:26.693 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:26.696+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:26.771 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:26.774+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:26.790 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.790 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:26.886 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:26.889+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:26.973 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:26.976+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:26.983 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:26.983 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:26.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:27.079 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:27.082+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:27.175 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:27.178+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:27.180 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.180 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.180 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 0.308211 (4ee6) 2024-04-24T15:59:27.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:27.276 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:27.278+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:27.375 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.375 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.376 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:27.379+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:27.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:27.477 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:27.480+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:27.522 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.523 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:27.578 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:27.581+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:27.657 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.657 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:27.679 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:27.682+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:27.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:27.779 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:27.782+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:27.790 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.790 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:27.884 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:27.887+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:27.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:27.950 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:27.965 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:27.965 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:27.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:27.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:27.989 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:28.008 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:28.016 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.019+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:28.028 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:28.082 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:28.085+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:28.139 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:28.139 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:28.139 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:28.139 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:28.139 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:28.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:28.183 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:28.186+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:28.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T15:59:28.285 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.287+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:28.313 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:28.313 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:28.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:28.313 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:28.313 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:28.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:28.406 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.409+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:28.485 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:28.485 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:28.486 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:28.486 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:28.487 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:28.489+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:28.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:28.588 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:28.591+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:28.658 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:28.659 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:28.659 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:28.659 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:28.659 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:28.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:28.712 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.714+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:28.789 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:28.792+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:28.792 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:28.792 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:28.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:28.890 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.893+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:28.950 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:28.951 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:28.951 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:28.951 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:28.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:28.990 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:28.993+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:29.091 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:29.094+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:29.100 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:29.100 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:29.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:29.192 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:29.195+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:29.194 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T15:59:29.252 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:29.253 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:29.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:29.253 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:29.253 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:29.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:29.293 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:29.296+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:29.394 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:29.397+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:29.403 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:29.403 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:29.403 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:29.403 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:29.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:29.495 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:29.498+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:29.554 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:29.554 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:29.554 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:29.554 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:29.554 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:29.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:29.595 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:29.598+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:29.696 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:29.699+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.717012244000003, 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.695766157000001, 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.681583895000003, 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.760 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.668086119000002, 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:29.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:29.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.66757681, 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:29.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:29.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.654247527999999, 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:29.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.65374181, 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:29.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:29.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:29.776 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.632184682000002, 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:29.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.485917112999999, 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.484663283000003, 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:29.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:29.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.481558104999998, 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:29.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:29.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:29.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.481120728, 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:29.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:29.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.480680976000002, 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.163880915, 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:29.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.163341703, 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:29.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.15970471, 2024-04-24T15:59:29.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.158509334000001, 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:29.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:29.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:29.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.158099038000003, 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:29.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:29.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.718359612, 2024-04-24T15:59:29.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:29.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.715754787999998, 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:29.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:29.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:29.830 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:29.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:29.832 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:29.834+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:29.932 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:29.935+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:30.017 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:30.018 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:30.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:30.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:30.073 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:30.076+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:30.134 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:30.137+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:30.170 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:30.170 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:30.170 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:30.170 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:30.170 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:30.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:30.228 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:30.235 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:30.238+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:30.248 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:30.269 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:30.290 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:30.310 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:30.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:30.335 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:30.339+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:30.364 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:30.364 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:30.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:30.436 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:30.439+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:30.512 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:30.512 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:30.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:30.563 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:30.566+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:30.638 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:30.641+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:30.728 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:30.729 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:30.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:30.740 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:30.741+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:30.840 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:30.842+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:30.892 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:30.892 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:30.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:30.945 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:30.948+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:31.024 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.025 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:31.076 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:31.078+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:31.142 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:31.144+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:31.199 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.199 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T15:59:31.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:31.254 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:31.257+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:31.334 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.334 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:31.344 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:31.345+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:31.445 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:31.447+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:31.507 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.507 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:31.545 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:31.547+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:31.646 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:31.648+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:31.680 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.681 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:31.747 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:31.749+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:31.864 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:31.867+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:31.949 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:31.952+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:31.971 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:31.971 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:31.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:32.050 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:32.052+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:32.121 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:32.121 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:32.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:32.151 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:32.154+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:32.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:32.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:32.195 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-24T15:59:32.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-24T15:59:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T15:59:32.268 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:32.270+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:32.320 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:32.320 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:32.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:32.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:32.353 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:32.355+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:32.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-24T15:59:32.454 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:32.457+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:32.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:32.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:32.554 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:32.557+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:32.583 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:32.605 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:32.633 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:32.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-24T15:59:32.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:32.655 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:32.657+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:32.672 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:32.681 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:32.681 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:32.681 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:32.681 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:32.682 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:32.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:32.756 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:32.758+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:32.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-24T15:59:32.874 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:32.876+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.850891668999999, 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.829645581999998, 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.815463319999999, 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.801965543999998, 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.801456235000003, 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.788126953000003, 2024-04-24T15:59:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.787621235000003, 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.766064106999998, 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.619796538000003, 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.618542708, 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.615437530000001, 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.615000152999997, 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.931 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.614560400999999, 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:32.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.297760340000004, 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.297221127999997, 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.293584135000003, 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.292388758999998, 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.291978463, 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.852239036999997, 2024-04-24T15:59:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.849634213000002, 2024-04-24T15:59:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:32.964 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:32.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:32.966 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:32.968+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:33.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-24T15:59:33.090 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:33.092+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:33.167 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:33.170+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:33.191 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:33.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:33.191 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:33.191 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:33.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:33.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-24T15:59:33.268 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:33.271+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:33.369 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:33.371+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:33.417 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:33.417 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:33.417 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:33.417 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:33.417 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:33.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:33.490 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:33.492+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:33.571 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:33.573+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:33.628 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:33.628 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:33.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:33.683 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:33.686+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:33.773 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:33.775+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:33.775 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:33.775 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:33.775 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:33.775 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:33.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:33.874 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:33.876+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:33.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:33.975 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:33.977+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:34.076 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:34.079+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:34.083 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:34.083 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:34.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.989862565000003, 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.982661401000001, 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:34.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:34.094 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.982604015, 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.099 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.82551969, 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.100 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.82544034, 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.105 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:34.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.825134869000003, 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:34.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:34.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:34.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.115 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.825113727000002, 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:34.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.825019830000002, 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:34.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:34.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.825007765999999, 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.126 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:34.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.824990341000003, 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:34.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:34.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:34.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:34.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.824924570999997, 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.136 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.824914102999998, 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.141 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:34.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:34.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:34.146 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499392782000001, 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499295625000002, 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:34.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:34.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.156 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499273381999998, 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499239455000001, 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499123230000002, 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499096409000003, 2024-04-24T15:59:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:34.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499074350999997, 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:34.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:34.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:34.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:34.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.183 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.499008605999997, 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:34.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:34.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:34.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:34.189 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:34.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:34.191 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:34.193+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:34.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:59:34.292 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:34.293+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:34.355 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:34.356 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:34.356 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:34.356 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:34.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:34.393 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:34.395+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:34.488 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:34.488 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:34.488 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:34.488 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:34.488 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:34.493 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:34.496+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:34.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:34.594 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:34.596+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:34.661 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:34.661 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:34.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:34.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:34.715 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:34.717+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:34.795 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:34.795 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:34.796 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:34.796 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:34.796 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:34.798+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:34.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:34.846 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:34.866 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:34.885 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:34.897 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:34.900+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:34.904 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:34.924 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:34.943 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:34.971 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:34.972 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:34.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:34.998 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:35.000+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:35.099 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:35.102+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:35.144 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:35.144 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:35.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:35.200 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:35.202+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:35.217 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:59:35.301 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:35.304+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:35.307 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:35.307 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:35.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:35.402 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:35.404+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:35.455 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:35.456 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:35.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:35.508 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:35.510+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:35.604 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:35.606+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:35.649 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:35.649 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:35.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:35.744 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:35.746+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:35.806 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:35.807+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:35.842 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:35.843 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:35.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:35.906 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:35.908+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:36.007 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:36.009+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:36.035 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.036 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:36.107 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:36.110+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:36.208 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:36.210+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:36.230 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.230 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:36.326 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:36.328+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:36.409 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:36.412+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:36.424 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.424 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:36.519 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:36.521+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:36.611 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:36.613+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:36.616 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.616 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:36.712 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:36.714+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:36.810 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.810 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.811 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:36.813+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:36.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:36.913 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:36.915+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:36.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:36.958 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:36.958 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:36.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:37.013 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:37.016+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:37.090 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:37.090 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:37.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:37.113 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:37.116+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:37.118 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:37.137 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:37.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:37.175 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:37.195 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:37.214 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:37.217+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:37.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:37.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T15:59:37.268 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:37.268 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:37.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:37.268 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.174764814, 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:37.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:37.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.167563649999998, 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.274 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:37.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.275 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:37.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:37.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.167506263999996, 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.010421938999997, 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.010342588999997, 2024-04-24T15:59:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.010037118, 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.010015975999998, 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:37.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:37.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.009922078999999, 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.308 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.009910015000003, 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.313 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:37.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:37.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.00989259, 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:37.318 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:37.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:37.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.009826820000001, 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.324 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:37.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:37.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:37.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.009816352000001, 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:37.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:37.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:37.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:37.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:37.334 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.684295030999998, 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:37.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:37.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:37.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:37.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.684197873999999, 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.340 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:37.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:37.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.684175631000002, 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.345 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:37.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.684141703999998, 2024-04-24T15:59:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:37.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:37.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.684025478999999, 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:37.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:37.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:37.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.683998658, 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:37.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:37.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:37.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.365 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.683976600000001, 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:37.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.683910855000001, 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:37.371 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:37.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:37.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:37.376 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:37.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:37.378 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:37.379+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:37.478 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:37.480+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:37.543 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:37.543 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:37.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:37.543 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:37.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:37.596 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:37.599+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:37.676 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:37.676 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:37.676 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:37.676 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:37.676 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:37.680 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:37.682+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:37.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:37.781 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:37.784+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:37.850 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:37.850 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:37.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:37.850 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:37.850 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:37.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:37.901 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:37.904+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:37.983 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:37.986+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:38.024 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:38.025 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:38.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:38.118 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:38.121+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:38.185 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:38.187+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:38.197 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:38.197 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:38.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:38.223 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T15:59:38.285 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:38.287+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:38.371 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:38.371 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:38.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:38.371 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-04-24T15:59:38.371 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:38.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:38.425 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:38.427+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:38.448 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:38.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:38.449 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T15:59:38.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T15:59:38.494 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:38.496+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:38.531 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:38.532 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:38.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:38.629 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:38.631+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:38.690 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:38.693+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:38.723 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:38.724 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:38.724 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:38.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:38.790 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:38.793+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:59:38.806 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 39, 2024-04-24T15:59:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c4fc9996-5ff9-4e71-b213-9020967fae1a", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-04-24T15:58:30.292182+0000", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-04-24T15:59:38.215829+0000", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-04-24T15:58:41.011852+0000", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-04-24T15:58:32.987203+0000", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5806080, 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-04-24T15:59:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 3, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 6, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "quincy", 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2024-04-24T15:59:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.444445+0000", 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2024-04-24T15:59:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "18", 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T15:59:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 16, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [ 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: 3 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": 3, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": 3, 2024-04-24T15:59:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-04-24T15:59:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.808270+0000", 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 1, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 1, 2024-04-24T15:59:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 1, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 1, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 1, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "14", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T15:59:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T15:59:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 3, 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cache", 2024-04-24T15:59:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:48.277449+0000", 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 9, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,incomplete_clones", 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 4, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 4, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 4, 2024-04-24T15:59:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 4, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 4, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "22", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 16, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": 1, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "writeback", 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 250, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bloom", 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "target_size": 0, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "seed": 0 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 60, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 8, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [ 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "51c39890-e79b-4030-a4a0-8c98ec2018ba", 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 15, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-04-24T15:59:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/41951", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/41951", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6819/41951", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/41951", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "1fb2523f-db36-46d9-80c4-dc08ac530daa", 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 34, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2024-04-24T15:59:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/41952", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/41952", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/41952", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/41952", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7d4dd13f-26d8-4c39-a6a6-020e91634f4e", 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0.308197021484375, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6814/41953", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6818/41953", 2024-04-24T15:59:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/41953", 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/41953", 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "03ed1eb6-8fa9-497c-b769-2743acc0e6aa", 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 12, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6801", 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2024-04-24T15:59:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/41923", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6807/41923", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6815/41923", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6811/41923", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f7ff11ec-62d6-44f3-bea1-7ff62d280e2f", 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 34, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6817/41924", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6819/41924", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6823/41924", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6821/41924", 2024-04-24T15:59:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "423ca5ac-f6a3-496e-a4c5-9ac205e2e474", 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 34, 2024-04-24T15:59:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6800", 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6802/41925", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6806/41925", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6814/41925", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6810/41925", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.171683+0000", 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.085846+0000", 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T15:59:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.118237+0000", 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.172469+0000", 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.139953+0000", 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.084518+0000", 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": {}, 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-04-24T15:59:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-04-24T15:59:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:38.862 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:38.863 INFO:tasks.thrashosds.thrasher:j is {'epoch': 39, 'fsid': 'c4fc9996-5ff9-4e71-b213-9020967fae1a', 'created': '2024-04-24T15:58:30.292182+0000', 'modified': '2024-04-24T15:59:38.215829+0000', 'last_up_change': '2024-04-24T15:58:41.011852+0000', 'last_in_change': '2024-04-24T15:58:32.987203+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-24T15:58:41.444445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-24T15:58:41.808270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T15:58:48.277449+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '51c39890-e79b-4030-a4a0-8c98ec2018ba', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 41951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 41951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6817', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 41951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6813', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 41951}]}, 'public_addr': '172.21.15.32:6809/41951', 'cluster_addr': '172.21.15.32:6811/41951', 'heartbeat_back_addr': '172.21.15.32:6819/41951', 'heartbeat_front_addr': '172.21.15.32:6815/41951', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fb2523f-db36-46d9-80c4-dc08ac530daa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 41952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 41952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 41952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 41952}]}, 'public_addr': '172.21.15.32:6801/41952', 'cluster_addr': '172.21.15.32:6803/41952', 'heartbeat_back_addr': '172.21.15.32:6807/41952', 'heartbeat_front_addr': '172.21.15.32:6805/41952', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7d4dd13f-26d8-4c39-a6a6-020e91634f4e', 'up': 1, 'in': 1, 'weight': 0.308197021484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6814', 'nonce': 41953}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6818', 'nonce': 41953}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 41953}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 41953}]}, 'public_addr': '172.21.15.32:6814/41953', 'cluster_addr': '172.21.15.32:6818/41953', 'heartbeat_back_addr': '172.21.15.32:6823/41953', 'heartbeat_front_addr': '172.21.15.32:6821/41953', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '03ed1eb6-8fa9-497c-b769-2743acc0e6aa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 12, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6803', 'nonce': 41923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6807', 'nonce': 41923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6815', 'nonce': 41923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6811', 'nonce': 41923}]}, 'public_addr': '172.21.15.86:6803/41923', 'cluster_addr': '172.21.15.86:6807/41923', 'heartbeat_back_addr': '172.21.15.86:6815/41923', 'heartbeat_front_addr': '172.21.15.86:6811/41923', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7ff11ec-62d6-44f3-bea1-7ff62d280e2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6816', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6817', 'nonce': 41924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6818', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6819', 'nonce': 41924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6822', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6823', 'nonce': 41924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6820', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6821', 'nonce': 41924}]}, 'public_addr': '172.21.15.86:6817/41924', 'cluster_addr': '172.21.15.86:6819/41924', 'heartbeat_back_addr': '172.21.15.86:6823/41924', 'heartbeat_front_addr': '172.21.15.86:6821/41924', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '423ca5ac-f6a3-496e-a4c5-9ac205e2e474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6802', 'nonce': 41925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6806', 'nonce': 41925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6814', 'nonce': 41925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6810', 'nonce': 41925}]}, 'public_addr': '172.21.15.86:6802/41925', 'cluster_addr': '172.21.15.86:6806/41925', 'heartbeat_back_addr': '172.21.15.86:6814/41925', 'heartbeat_front_addr': '172.21.15.86:6810/41925', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.171683+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.085846+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.118237+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.172469+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.139953+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.084518+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T15:59:38.863 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T15:59:38.904 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:38.906+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:38.914 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:38.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:39.009 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:39.011+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:39.093 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:39.095+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:39.108 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:39.108 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:39.108 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:39.108 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:39.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:39.206 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:39.208+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:39.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:39.296 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:39.299+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:39.305 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:39.305 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:39.305 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:39.305 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:39.305 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:39.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:39.396 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:39.398+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:39.408 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:39.447 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:39.471 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:39.492 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:39.497 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:39.500+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:39.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:39.514 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:39.533 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:39.598 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:39.601+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:39.652 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:39.652 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:39.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:39.707 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:39.709+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:39.800 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:39.802+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:39.802 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:39.802 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:39.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:39.901 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:39.904+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:39.996 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:39.996 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.002 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:40.005+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:40.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:40.103 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:40.105+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:40.143 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.143 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:40.204 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:40.206+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:40.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T15:59:40.277 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.277 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:40.328 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:40.330+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:40.406 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:40.408+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:40.451 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.451 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:40.508 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:40.509+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:40.584 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.585 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:40.608 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:40.610+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:40.709 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:40.711+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:40.757 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.758 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:40.810 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:40.811+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:40.911 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:40.914+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:40.931 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:40.932 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:40.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:41.012 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:41.014+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:41.066 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:41.066 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:41.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:41.113 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:41.115+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:41.214 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:41.216+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T15:59:41.239 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:41.239 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:41.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:41.315 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:41.317+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:41.372 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:41.372 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:41.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:41.426 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:41.428+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:41.517 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:41.519+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:41.531 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:41.531 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:41.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:41.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:41.618 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:41.620+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:41.718 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:41.720+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:41.728 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:41.728 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:41.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:41.743 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:41.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:41.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:41.806 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:41.819 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:41.821+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:41.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:41.847 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:41.882 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:41.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:41.934 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:41.936+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:42.021 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:42.023+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:42.075 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:42.075 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:42.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:42.128 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:42.130+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:42.223 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:42.225+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:42.272 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:42.272 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:42.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:42.324 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:42.326+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:42.424 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:42.426+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:1 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'1, mlcod=0'0)", 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.436066752999999, 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015690369999999999, 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032608+0000", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032607+0000", 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032614+0000", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1130000000000003e-06 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032623+0000", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0729999999999994e-06 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032631+0000", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2860000000000002e-06 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032669+0000", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8111999999999997e-05 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.032691+0000", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2713000000000001e-05 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034134+0000", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001442135 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.034177+0000", 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3287000000000001e-05 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:20 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'10, mlcod=13'8)", 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.414820665999997, 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001071462, 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053854+0000", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053853+0000", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053964+0000", 2024-04-24T15:59:42.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110504 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053975+0000", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1035e-05 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.053980+0000", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8960000000000002e-06 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054013+0000", 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3788999999999998e-05 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054028+0000", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4404e-05 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054882+0000", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085468800000000004 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.054925+0000", 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2982999999999998e-05 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.474 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:35 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'21, mlcod=13'15)", 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.400638403999999, 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010656769999999999, 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068036+0000", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068035+0000", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068092+0000", 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6718999999999997e-05 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068107+0000", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4973e-05 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068113+0000", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068150+0000", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6437999999999999e-05 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.068164+0000", 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4048e-05 2024-04-24T15:59:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069079+0000", 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091567000000000005 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.069102+0000", 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:54 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.387140627999997, 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011077120000000001, 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081534+0000", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081550+0000", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081559+0000", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6760000000000002e-06 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081565+0000", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7910000000000001e-06 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081610+0000", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4634e-05 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.081627+0000", 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7626999999999999e-05 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082602+0000", 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097490800000000002 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082642+0000", 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9475999999999999e-05 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:55 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'33, mlcod=13'29)", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.386631319000003, 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011025399999999999, 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082043+0000", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082042+0000", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082107+0000", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4574999999999998e-05 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082116+0000", 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3719999999999994e-06 2024-04-24T15:59:42.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082121+0000", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5730000000000001e-06 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082168+0000", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6995e-05 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.082190+0000", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2203000000000001e-05 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083106+0000", 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091563300000000001 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.083146+0000", 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0095000000000002e-05 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:68 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.373302037000002, 2024-04-24T15:59:42.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011679710000000001, 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095373+0000", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095372+0000", 2024-04-24T15:59:42.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095377+0000", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095386+0000", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7260000000000008e-06 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095390+0000", 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.053e-06 2024-04-24T15:59:42.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095426+0000", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5777e-05 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095446+0000", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0519000000000001e-05 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096498+0000", 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001051279 2024-04-24T15:59:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096541+0000", 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2763999999999999e-05 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:69 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.372796319000003, 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010828179999999999, 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095878+0000", 2024-04-24T15:59:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095937+0000", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9422000000000001e-05 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095953+0000", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.59e-05 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095958+0000", 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-04-24T15:59:42.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.095997+0000", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9718000000000002e-05 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096018+0000", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0947000000000001e-05 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096925+0000", 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090700799999999995 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.096961+0000", 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5744999999999997e-05 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:84 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.351239190999998, 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001052975, 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117435+0000", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117491+0000", 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6611999999999999e-05 2024-04-24T15:59:42.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117500+0000", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2889999999999998e-06 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117504+0000", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-06 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117546+0000", 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1425999999999999e-05 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.117558+0000", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2771e-05 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118467+0000", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090905000000000001 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.118488+0000", 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0910000000000001e-05 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:112 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'74, mlcod=13'66)", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.204971622000002, 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001231383, 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263703+0000", 2024-04-24T15:59:42.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263702+0000", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263767+0000", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4972000000000001e-05 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263776+0000", 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7420000000000003e-06 2024-04-24T15:59:42.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263780+0000", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1280000000000001e-06 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263818+0000", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7429999999999999e-05 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.263838+0000", 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264906+0000", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106796 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264934+0000", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8585000000000002e-05 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:114 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'76, mlcod=13'71)", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:42.501 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.203717791999999, 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011141829999999999, 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264957+0000", 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264956+0000", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264963+0000", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264972+0000", 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4490000000000005e-06 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.264976+0000", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8229999999999997e-06 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265011+0000", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.265033+0000", 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1460000000000001e-05 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266051+0000", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001018342 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.266071+0000", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9916e-05 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:118 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.200612614000001, 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001397782, 2024-04-24T15:59:42.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268062+0000", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268061+0000", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268138+0000", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6555000000000001e-05 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268148+0000", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268152+0000", 2024-04-24T15:59:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268189+0000", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6772e-05 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268211+0000", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2259000000000002e-05 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269418+0000", 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012063429999999999 2024-04-24T15:59:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269460+0000", 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2249999999999997e-05 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:119 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'81, mlcod=13'78)", 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.200175237000003, 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001387087, 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268499+0000", 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268566+0000", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7158999999999999e-05 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268574+0000", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.106e-06 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268578+0000", 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.194e-06 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268619+0000", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1152000000000002e-05 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268640+0000", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269865+0000", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001224852 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269886+0000", 2024-04-24T15:59:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1639000000000001e-05 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:120 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'82, mlcod=13'78)", 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.199735484999998, 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001072368, 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268939+0000", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.268938+0000", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269014+0000", 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5525999999999995e-05 2024-04-24T15:59:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269022+0000", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1410000000000005e-06 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269026+0000", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6100000000000002e-06 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269060+0000", 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4690999999999997e-05 2024-04-24T15:59:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269081+0000", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0738999999999999e-05 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.269993+0000", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091220799999999996 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.270011+0000", 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8196e-05 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.516 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:130 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'88, mlcod=13'85)", 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.882935424000003, 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00106364, 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585739+0000", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585738+0000", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585747+0000", 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6410000000000008e-06 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585758+0000", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0988999999999999e-05 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585763+0000", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585799+0000", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599000000000003e-05 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.585819+0000", 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0233000000000001e-05 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586769+0000", 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094958399999999995 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586803+0000", 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4155000000000001e-05 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:131 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'89, mlcod=13'85)", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.882396212000003, 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001039938, 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:42.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586278+0000", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586352+0000", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3985000000000004e-05 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586362+0000", 2024-04-24T15:59:42.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0162e-05 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586366+0000", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.335e-06 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586407+0000", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0883999999999997e-05 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.586419+0000", 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1855000000000001e-05 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587280+0000", 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086155400000000005 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.587318+0000", 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7945999999999999e-05 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:137 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'90)", 2024-04-24T15:59:42.524 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.878759219000003, 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012243149999999999, 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589915+0000", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.589988+0000", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3663000000000003e-05 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590001+0000", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2996999999999999e-05 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590006+0000", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2749999999999997e-06 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590050+0000", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.590082+0000", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2240000000000003e-05 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591078+0000", 2024-04-24T15:59:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099567699999999993 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591140+0000", 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1946000000000006e-05 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:139 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.877563842999997, 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010671389999999999, 2024-04-24T15:59:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591111+0000", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591110+0000", 2024-04-24T15:59:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591116+0000", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591130+0000", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3973e-05 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591134+0000", 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3669999999999999e-06 2024-04-24T15:59:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591173+0000", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8312999999999997e-05 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591194+0000", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1654999999999998e-05 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592142+0000", 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094821100000000002 2024-04-24T15:59:42.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592178+0000", 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5485999999999999e-05 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:140 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'93)", 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.877153546999999, 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011662440000000001, 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591521+0000", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591520+0000", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591594+0000", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374000000000006e-05 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591601+0000", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9460000000000001e-06 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591606+0000", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9710000000000003e-06 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591639+0000", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3133999999999998e-05 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.591660+0000", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1562000000000001e-05 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592655+0000", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099443199999999995 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:43.592687+0000", 2024-04-24T15:59:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:158 2.0 e13/12 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'107)", 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.437414121000003, 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013345060000000001, 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031260+0000", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031337+0000", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7344999999999999e-05 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031347+0000", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6560000000000004e-06 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031353+0000", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7389999999999996e-06 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031381+0000", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8051e-05 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.031401+0000", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619e-05 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032559+0000", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011579660000000001 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.032595+0000", 2024-04-24T15:59:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5590000000000003e-05 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4171.0:161 2.0 e13/12 2:30306672:devicehealth::main.db.0000000000000000:head v 13'115, mlcod=13'112)", 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.434809297000001, 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00108647, 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033865+0000", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033949+0000", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4530000000000006e-05 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033962+0000", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.24e-05 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.033967+0000", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7540000000000003e-06 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034008+0000", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0497000000000002e-05 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034028+0000", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9962999999999998e-05 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034921+0000", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089350699999999998 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:58:44.034952+0000", 2024-04-24T15:59:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0454e-05 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:42.544 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:42.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:42.546 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:42.547+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:42.646 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:42.648+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:42.688 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:42.688 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:42.688 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:42.688 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:42.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:42.784 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:42.786+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:42.848 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:42.850+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:42.880 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:42.881 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:42.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:42.976 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:42.978+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:43.050 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:43.052+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:43.072 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:43.073 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:43.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:43.073 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:43.073 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:43.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:43.151 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:43.152+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:43.221 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:43.221 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:43.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:43.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T15:59:43.290 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:43.292+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:43.352 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:43.354+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:43.396 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:43.396 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:43.396 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:43.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:43.453 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:43.455+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:43.530 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:43.530 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.436853839999998, 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:43.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.429652676000003, 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.536 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:43.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:43.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:43.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.429595290000002, 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.541 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:43.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:43.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:43.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.546 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.272510965000002, 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.551 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.272431615000002, 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.272126143999998, 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:43.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.272105002000004, 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:43.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:43.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:43.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:43.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.272011104999997, 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.567 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.271999041000001, 2024-04-24T15:59:43.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:43.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.271981615999998, 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.271915845999999, 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:43.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:43.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.587 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.271905378, 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:43.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:43.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946384057000003, 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.593 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946286899999997, 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.598 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:43.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:43.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:43.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:43.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946264657, 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.603 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946230730000003, 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946114504999997, 2024-04-24T15:59:43.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:43.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:43.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:43.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:43.618 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946087683999998, 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.619 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.946065625999999, 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.625 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:43.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:43.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:43.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:43.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 59.945999880999999, 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:43.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:43.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:43.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:43.637 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:43.640+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:43.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:43.739 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:43.740+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:43.762 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:43.763 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:43.763 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:43.763 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:43.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:43.856 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:43.858+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:43.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:43.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:43.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:43.865 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T15:59:43.865 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-04-24T15:59:43.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T15:59:43.936 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:43.936 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:43.936 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:43.936 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:43.936 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:43.940 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:43.942+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:43.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:44.028 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:44.041 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:44.043+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:44.048 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:44.067 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:44.087 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:44.107 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:44.110 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.110 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:44.126 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:44.128 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:44.161 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:44.163+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:44.241 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T15:59:44.242 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:44.244+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:44.289 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.289 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:44.383 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:44.385+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:44.444 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:44.446+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:44.463 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.463 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:44.556 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:44.558+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:44.636 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.636 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:44.649 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:44.648+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:44.747 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:44.748+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:44.798 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.799 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:44.848 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:44.850+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:44.948 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:44.950+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:44.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:45.049 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:45.051+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:45.142 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:45.142 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:45.150 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:45.152+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:45.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:45.243 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T15:59:45.251 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:45.253+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:45.339 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:45.339 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:45.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:45.352 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:45.354+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:45.454 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:45.456+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:45.490 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:45.490 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:45.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:45.555 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:45.557+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:Note: switching to '330067dbce47655d1f762ef14d546d2fde16a6a9'. 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T15:59:45.619 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: git switch -c 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr:Or undo this operation with: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: git switch - 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-04-24T15:59:45.620 INFO:tasks.workunit.client.0.smithi122.stderr:HEAD is now at 330067dbce4 Merge PR #54172 into wip-yuriw-testing-20240423.151325-quincy 2024-04-24T15:59:45.625 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T15:59:45.640 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:45.640 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:45.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:45.690 INFO:tasks.workunit.client.0.smithi122.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T15:59:45.692 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T15:59:45.692 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T15:59:45.693 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:45.695+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:45.757 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:45.759+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:45.833 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:45.834 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:45.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:45.857 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:45.859+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:45.959 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:45.961+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:46.027 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:46.027 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:46.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:46.060 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:46.062+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:46.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:46.159 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T15:59:46.180 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:46.180 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:46.180 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:46.182+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:46.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:46.215 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T15:59:46.253 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T15:59:46.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:59:46.255 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T15:59:46.255 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T15:59:46.261 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:46.263+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:46.291 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T15:59:46.295 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-24T15:59:46.295 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T15:59:46.300 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:46.320 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:46.329 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:46.329 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:46.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:46.340 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:46.352 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-04-24T15:59:46.353 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-04-24T15:59:46.354 DEBUG:teuthology.orchestra.run.smithi122:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=330067dbce47655d1f762ef14d546d2fde16a6a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-04-24T15:59:46.359 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:46.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:46.380 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:46.382+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:46.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:46.413 INFO:tasks.workunit.client.0.smithi122.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ TIMEOUT=300 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ MAX_TIMEOUT=15 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ PG_NUM=4 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ TMPDIR=/tmp 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ type xmlstarlet 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ XMLSTARLET=xmlstarlet 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:+++ uname 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' Linux = FreeBSD ']' 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ SED=sed 2024-04-24T15:59:46.416 INFO:tasks.workunit.client.0.smithi122.stderr:++ AWK=awk 2024-04-24T15:59:46.417 INFO:tasks.workunit.client.0.smithi122.stderr:+++ stty -a 2024-04-24T15:59:46.417 INFO:tasks.workunit.client.0.smithi122.stderr:+++ head -1 2024-04-24T15:59:46.417 INFO:tasks.workunit.client.0.smithi122.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-04-24T15:59:46.418 INFO:tasks.workunit.client.0.smithi122.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-24T15:59:46.418 INFO:tasks.workunit.client.0.smithi122.stderr:++ termwidth= 2024-04-24T15:59:46.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-04-24T15:59:46.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ DIFFCOLOPTS='-y ' 2024-04-24T15:59:46.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ KERNCORE=kernel.core_pattern 2024-04-24T15:59:46.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ EXTRA_OPTS= 2024-04-24T15:59:46.422 INFO:tasks.workunit.client.0.smithi122.stderr:++ test '' = TESTS 2024-04-24T15:59:46.423 INFO:tasks.workunit.client.0.smithi122.stderr:+ POOL=rbd 2024-04-24T15:59:46.426 INFO:tasks.workunit.client.0.smithi122.stderr:+ ANOTHER_POOL=new_default_pool37497 2024-04-24T15:59:46.426 INFO:tasks.workunit.client.0.smithi122.stderr:+ NS=ns 2024-04-24T15:59:46.426 INFO:tasks.workunit.client.0.smithi122.stderr:+ IMAGE=testrbdnbd37497 2024-04-24T15:59:46.426 INFO:tasks.workunit.client.0.smithi122.stderr:+ SIZE=64 2024-04-24T15:59:46.426 INFO:tasks.workunit.client.0.smithi122.stderr:+ DATA= 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ setup 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns x 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -e CMakeCache.txt ']' 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo echo test sudo 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:46.427 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:++ which echo 2024-04-24T15:59:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/echo 2024-04-24T15:59:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-04-24T15:59:46.435 INFO:tasks.workunit.client.0.smithi122.stdout:test sudo 2024-04-24T15:59:46.436 INFO:tasks.workunit.client.0.smithi122.stderr:+ trap cleanup INT TERM EXIT 2024-04-24T15:59:46.436 INFO:tasks.workunit.client.0.smithi122.stderr:++ mktemp -d 2024-04-24T15:59:46.437 INFO:tasks.workunit.client.0.smithi122.stderr:+ TEMPDIR=/tmp/tmp.P3d5b6KQ4E 2024-04-24T15:59:46.437 INFO:tasks.workunit.client.0.smithi122.stderr:+ DATA=/tmp/tmp.P3d5b6KQ4E/data 2024-04-24T15:59:46.437 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/dev/urandom of=/tmp/tmp.P3d5b6KQ4E/data bs=1M count=64 2024-04-24T15:59:46.464 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:46.465+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:46.502 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:46.503 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:46.503 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:46.503 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:46.503 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:46.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:46.565 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:46.567+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:46.666 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:46.668+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:46.679 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:46.680 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:46.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:46.680 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:46.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.680 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.585560610999998, 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069337449999999998, 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094779+0000", 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094778+0000", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094781+0000", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5110000000000002e-06 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094785+0000", 2024-04-24T15:59:46.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-06 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.094788+0000", 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7149999999999998e-06 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098519+0000", 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037314359999999999 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098613+0000", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3029000000000007e-05 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.098750+0000", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013753699999999999 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.099595+0000", 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084460100000000005 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101654+0000", 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002059066 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101684+0000", 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0202999999999999e-05 2024-04-24T15:59:46.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101712+0000", 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8475000000000001e-05 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:73 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.578359446999997, 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059830370000000001, 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.687 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101980+0000", 2024-04-24T15:59:46.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.101979+0000", 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102018+0000", 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9002999999999999e-05 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102026+0000", 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.622e-06 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102030+0000", 2024-04-24T15:59:46.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4850000000000003e-06 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102932+0000", 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090145799999999995 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103018+0000", 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6490999999999996e-05 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103113+0000", 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4549999999999994e-05 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104177+0000", 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010639790000000001 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107913+0000", 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037357430000000001 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107943+0000", 2024-04-24T15:59:46.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9841000000000001e-05 2024-04-24T15:59:46.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107963+0000", 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0360000000000002e-05 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.698 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:74 2.0 2:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.578302061000002, 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059761049999999998, 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102037+0000", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102041+0000", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1570000000000002e-06 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102050+0000", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1740000000000006e-06 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.102054+0000", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9430000000000002e-06 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103374+0000", 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013199970000000001 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103457+0000", 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2659999999999998e-05 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.103593+0000", 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013654300000000001 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.104565+0000", 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097189399999999995 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107969+0000", 2024-04-24T15:59:46.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034035139999999998 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.107991+0000", 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2408e-05 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.108013+0000", 2024-04-24T15:59:46.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2302000000000001e-05 2024-04-24T15:59:46.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.706 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.707 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.707 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.707 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.421217736000003, 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049709940000000003, 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.708 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.709 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.709 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106 2024-04-24T15:59:46.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.709 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259122+0000", 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259121+0000", 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:46.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259183+0000", 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2067000000000001e-05 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259191+0000", 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259196+0000", 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-04-24T15:59:46.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260052+0000", 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085558699999999999 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260102+0000", 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9972999999999997e-05 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260185+0000", 2024-04-24T15:59:46.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3067999999999998e-05 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261788+0000", 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016034210000000001 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264016+0000", 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002227728 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264045+0000", 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.94e-05 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264093+0000", 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.421138386000003, 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049583020000000004, 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 107 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.720 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259201+0000", 2024-04-24T15:59:46.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259200+0000", 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259203+0000", 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7300000000000001e-06 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259212+0000", 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0119999999999994e-06 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259216+0000", 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.856e-06 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260638+0000", 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014215740000000001 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260723+0000", 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4963000000000006e-05 2024-04-24T15:59:46.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.260849+0000", 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012654100000000001 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.261822+0000", 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000973288 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264101+0000", 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022785280000000002 2024-04-24T15:59:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264126+0000", 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4691000000000001e-05 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264159+0000", 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3509000000000001e-05 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420832914999998, 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005731056, 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259506+0000", 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259509+0000", 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3510000000000002e-06 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259515+0000", 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0390000000000001e-06 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259519+0000", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.262976+0000", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034569879999999998 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263053+0000", 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6783e-05 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263184+0000", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130724 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265148+0000", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019645330000000001 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265176+0000", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8229e-05 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265204+0000", 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7209999999999999e-05 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265237+0000", 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3787e-05 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420811772999997, 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006893677, 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259527+0000", 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259601+0000", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3759999999999999e-05 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259610+0000", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1619999999999997e-06 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259615+0000", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.972e-06 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263479+0000", 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038644080000000002 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263535+0000", 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5998999999999997e-05 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.263616+0000", 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0282000000000001e-05 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265172+0000", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001556308 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266342+0000", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011698279999999999 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266386+0000", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4422999999999998e-05 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266421+0000", 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4989e-05 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:46.744 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420717875999998, 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069437780000000003, 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 113 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.745 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259621+0000", 2024-04-24T15:59:46.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259624+0000", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9450000000000002e-06 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259628+0000", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0840000000000002e-06 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259630+0000", 2024-04-24T15:59:46.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4219999999999999e-06 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264376+0000", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047455149999999996 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264432+0000", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5618999999999997e-05 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264512+0000", 2024-04-24T15:59:46.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0161999999999996e-05 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266430+0000", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001917787 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266477+0000", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7865999999999999e-05 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266527+0000", 2024-04-24T15:59:46.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9274000000000003e-05 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266565+0000", 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8464999999999998e-05 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420705812000001, 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070690140000000002, 2024-04-24T15:59:46.750 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 114 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259633+0000", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259636+0000", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2979999999999999e-06 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259639+0000", 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2559999999999998e-06 2024-04-24T15:59:46.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259641+0000", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0140000000000001e-06 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264765+0000", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005124661 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264812+0000", 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6220000000000001e-05 2024-04-24T15:59:46.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.264888+0000", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6630000000000003e-05 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266639+0000", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017508179999999999 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266646+0000", 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.516e-06 2024-04-24T15:59:46.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266675+0000", 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9465e-05 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266702+0000", 2024-04-24T15:59:46.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7361000000000001e-05 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420688386999998, 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074191580000000003, 2024-04-24T15:59:46.756 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 115 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:46.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259651+0000", 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259702+0000", 2024-04-24T15:59:46.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0924000000000002e-05 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259709+0000", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0199999999999997e-06 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259713+0000", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265276+0000", 2024-04-24T15:59:46.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055630039999999999 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265344+0000", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8435000000000005e-05 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265472+0000", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012762 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266709+0000", 2024-04-24T15:59:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001237394 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267021+0000", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031226099999999998 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267049+0000", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7855e-05 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267070+0000", 2024-04-24T15:59:46.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931999999999999e-05 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420622616999999, 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074896399999999997, 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.762 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 116 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259717+0000", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259716+0000", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259719+0000", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6730000000000001e-06 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259722+0000", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259724+0000", 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8959999999999999e-06 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265858+0000", 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061340149999999996 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.265939+0000", 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0891999999999997e-05 2024-04-24T15:59:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266081+0000", 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014209800000000001 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267080+0000", 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009989179999999999 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267126+0000", 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6050000000000001e-05 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267171+0000", 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4419999999999998e-05 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267206+0000", 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5528999999999998e-05 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.420612149, 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079241889999999999, 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259727+0000", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259729+0000", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4049999999999998e-06 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259732+0000", 2024-04-24T15:59:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0529999999999998e-06 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.259734+0000", 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266725+0000", 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069903869999999998 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266772+0000", 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999997e-05 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.266849+0000", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6611000000000002e-05 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267261+0000", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041248100000000002 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267575+0000", 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031424600000000001 2024-04-24T15:59:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267618+0000", 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2378e-05 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.267651+0000", 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3694e-05 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.095090827999996, 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122087, 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585248+0000", 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585321+0000", 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3344999999999997e-05 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585333+0000", 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.151e-05 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585338+0000", 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.164e-06 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587802+0000", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024644269999999999 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587849+0000", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6910999999999999e-05 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.587924+0000", 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5051999999999999e-05 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589597+0000", 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016732369999999999 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590279+0000", 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068114499999999997 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590338+0000", 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-05 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590371+0000", 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2786000000000001e-05 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094993670999997, 2024-04-24T15:59:46.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055003209999999999, 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585346+0000", 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585345+0000", 2024-04-24T15:59:46.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585349+0000", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2250000000000002e-06 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585358+0000", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5599999999999994e-06 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585362+0000", 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8929999999999998e-06 2024-04-24T15:59:46.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588558+0000", 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031958440000000002 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588629+0000", 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1125999999999996e-05 2024-04-24T15:59:46.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.588762+0000", 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013369799999999999 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590793+0000", 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020301939999999999 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590797+0000", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5569999999999997e-06 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590824+0000", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590846+0000", 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2419999999999999e-05 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094971428000001, 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059224259999999997, 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.788 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585368+0000", 2024-04-24T15:59:46.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585367+0000", 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585371+0000", 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-04-24T15:59:46.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585378+0000", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8020000000000003e-06 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585382+0000", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1049999999999997e-06 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589087+0000", 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037050490000000002 2024-04-24T15:59:46.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589134+0000", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6417e-05 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589207+0000", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3269000000000006e-05 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590888+0000", 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016812629999999999 2024-04-24T15:59:46.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591245+0000", 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035702599999999998 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591271+0000", 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5579e-05 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591290+0000", 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9536999999999999e-05 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094937500999997, 2024-04-24T15:59:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070211620000000001, 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 137 2024-04-24T15:59:46.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585402+0000", 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585401+0000", 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585471+0000", 2024-04-24T15:59:46.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9785999999999999e-05 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585479+0000", 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3920000000000003e-06 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585485+0000", 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589627+0000", 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041421080000000002 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589707+0000", 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0105999999999995e-05 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.589831+0000", 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123566 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592031+0000", 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022006679999999998 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592377+0000", 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000346044 2024-04-24T15:59:46.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592403+0000", 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.563e-05 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592423+0000", 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9911e-05 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094821275999998, 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069814639999999997, 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 138 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.806 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585518+0000", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585517+0000", 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585527+0000", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.842e-06 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585534+0000", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2010000000000002e-06 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585539+0000", 2024-04-24T15:59:46.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3229999999999999e-06 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590407+0000", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048685949999999999 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590454+0000", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590527+0000", 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2193999999999994e-05 2024-04-24T15:59:46.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592064+0000", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001537353 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592430+0000", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036619799999999998 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592470+0000", 2024-04-24T15:59:46.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0411000000000003e-05 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592499+0000", 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8986999999999999e-05 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094794454999999, 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070335320000000003, 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.811 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585545+0000", 2024-04-24T15:59:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585544+0000", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585551+0000", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585557+0000", 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5880000000000003e-06 2024-04-24T15:59:46.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585561+0000", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8460000000000001e-06 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590916+0000", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005355032 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.590961+0000", 2024-04-24T15:59:46.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5142999999999998e-05 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591041+0000", 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9098999999999996e-05 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592508+0000", 2024-04-24T15:59:46.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014671949999999999 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592530+0000", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2524999999999999e-05 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592558+0000", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7319000000000001e-05 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592578+0000", 2024-04-24T15:59:46.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.073e-05 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094772397, 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073432439999999996, 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 140 2024-04-24T15:59:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585567+0000", 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585566+0000", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585620+0000", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4082000000000002e-05 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585626+0000", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2809999999999999e-06 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585629+0000", 2024-04-24T15:59:46.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8200000000000001e-06 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591320+0000", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056912380000000004 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591366+0000", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6332999999999998e-05 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.591441+0000", 2024-04-24T15:59:46.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4851999999999994e-05 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592767+0000", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013264730000000001 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592865+0000", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7102999999999995e-05 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592890+0000", 2024-04-24T15:59:46.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5109e-05 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592910+0000", 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0457000000000001e-05 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.822 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4171.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.094706651999999, 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008125307, 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4171", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1655033652", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585633+0000", 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585632+0000", 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585636+0000", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680000000000001e-06 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585642+0000", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6289999999999998e-06 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.585645+0000", 2024-04-24T15:59:46.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4879999999999999e-06 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592077+0000", 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00643203 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592123+0000", 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5988000000000001e-05 2024-04-24T15:59:46.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.592194+0000", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0452999999999998e-05 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593132+0000", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093777699999999999 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593706+0000", 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00057452800000000004 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593729+0000", 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.249e-05 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:58:43.593758+0000", 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386e-05 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:46.828 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:46.830 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:46.830 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:46.831 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.291165 s, 230 MB/s 2024-04-24T15:59:46.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd namespace create rbd/ns 2024-04-24T15:59:46.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-04-24T15:59:46.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.P3d5b6KQ4E/data testrbdnbd37497 2024-04-24T15:59:46.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:46.833 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:46.831+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:46.930 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:46.932+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:46.997 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:46.997 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:46.997 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:46.997 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:47.031 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:47.033+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:47.129 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:47.129 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:47.130 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:47.130 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:47.130 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.132 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:47.134+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:47.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:47.233 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:47.235+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:47.249 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:2 3.0 e42/34 3:377edd40:::rbd_namespace:head v 42'1, mlcod=0'0)", 2024-04-24T15:59:47.305 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.763072+0000", 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.544503246, 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00055473100000000004, 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763072+0000", 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763072+0000", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763072+0000", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763078+0000", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0549999999999996e-06 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763087+0000", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8899999999999996e-06 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763101+0000", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3807e-05 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763132+0000", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1881000000000002e-05 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763145+0000", 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2661e-05 2024-04-24T15:59:47.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763592+0000", 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000446926 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.763627+0000", 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4860000000000002e-05 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:1 3.3 3:da3fcecf:::testrbdnbd37497.rbd:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.798139+0000", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.50943594599999997, 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001088171, 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 1 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798139+0000", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798139+0000", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798138+0000", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798143+0000", 2024-04-24T15:59:47.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3959999999999999e-06 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798150+0000", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1600000000000001e-06 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798155+0000", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7430000000000002e-06 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.798195+0000", 2024-04-24T15:59:47.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0720999999999999e-05 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799228+0000", 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010323050000000001 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:2 3.3 3:c336d817:::rbd_id.testrbdnbd37497:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.799395+0000", 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.50818081900000001, 2024-04-24T15:59:47.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00087442599999999998, 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 2 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799395+0000", 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799395+0000", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799393+0000", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799403+0000", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0156999999999999e-05 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799421+0000", 2024-04-24T15:59:47.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7873e-05 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799431+0000", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5119999999999997e-06 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.799497+0000", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6179000000000005e-05 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800269+0000", 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00077193800000000001 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.316 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(osd.4.0:0 3.3 e42/15 ondisk, result = 0)", 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.800820+0000", 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.50675577400000005, 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2440000000000004e-05, 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800820+0000", 2024-04-24T15:59:47.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800820+0000", 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800819+0000", 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800822+0000", 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0230000000000001e-06 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800830+0000", 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7700000000000001e-06 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800836+0000", 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3550000000000001e-06 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800857+0000", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0630999999999999e-05 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800862+0000", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.451e-06 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.800892+0000", 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9748999999999999e-05 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.320 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(osd.4.0:1 3.3 e42/15 ondisk, result = 0)", 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.801148+0000", 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.50642769099999996, 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3272e-05, 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801148+0000", 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801148+0000", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801147+0000", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801151+0000", 2024-04-24T15:59:47.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.473e-06 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801157+0000", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9229999999999999e-06 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801161+0000", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3830000000000002e-06 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801179+0000", 2024-04-24T15:59:47.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7481000000000001e-05 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801183+0000", 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2899999999999996e-06 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.801211+0000", 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8081000000000001e-05 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.324 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:10 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:47.004637+0000", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.30293851199999999, 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020721199999999999, 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004637+0000", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004637+0000", 2024-04-24T15:59:47.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004635+0000", 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004645+0000", 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.043e-05 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004663+0000", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.806e-05 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004690+0000", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6746000000000001e-05 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004749+0000", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8891999999999998e-05 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004844+0000", 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5284000000000005e-05 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:11 1.2 1:5cea7035:::rbd_info:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e42)", 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:47.004718+0000", 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.30285698599999999, 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020511300000000001, 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004718+0000", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004718+0000", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004718+0000", 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004721+0000", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.112e-06 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004729+0000", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8499999999999994e-06 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004734+0000", 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5090000000000001e-06 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004860+0000", 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012586800000000001 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004923+0000", 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3324999999999994e-05 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:12 1.2 1:5cea7035:::rbd_info:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e42)", 2024-04-24T15:59:47.331 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:47.004744+0000", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.302831605, 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00024350399999999999, 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004744+0000", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004744+0000", 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004743+0000", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004747+0000", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8369999999999999e-06 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004762+0000", 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4528999999999999e-05 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004767+0000", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4049999999999999e-06 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004938+0000", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000170977 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.004987+0000", 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9199000000000001e-05 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.335 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.335 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:47.337 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:47.338+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:47.437 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:47.438+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:47.500 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:47.500 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:47.500 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:47.501 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:47.551 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:47.553+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:47.638 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:47.640+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:47.674 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:47.674 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:47.674 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:47.674 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:47.674 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:47.768 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:47.770+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:47.840 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:47.842+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:3 3.2 e42/15 3:774f7988:ns::rbd_directory:head v 42'2, mlcod=0'0)", 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:46.765414+0000", 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.0857865609999999, 2024-04-24T15:59:47.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00058985000000000005, 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765414+0000", 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765414+0000", 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765413+0000", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765423+0000", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0720000000000001e-05 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765437+0000", 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3485e-05 2024-04-24T15:59:47.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765453+0000", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6569e-05 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765488+0000", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4894000000000002e-05 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765503+0000", 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4817000000000001e-05 2024-04-24T15:59:47.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.765977+0000", 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00047369300000000002 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:46.766003+0000", 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6678000000000001e-05 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.852 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:4 3.2 e42/15 3:5cea7035:::rbd_info:head v 42'3, mlcod=38'1)", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:47.005297+0000", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.84590303099999997, 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00063511000000000004, 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005297+0000", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005297+0000", 2024-04-24T15:59:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005296+0000", 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005304+0000", 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2849999999999995e-06 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005317+0000", 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2714000000000001e-05 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005323+0000", 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6659999999999996e-06 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005363+0000", 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0153e-05 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005381+0000", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8766e-05 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005885+0000", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000503656 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.005932+0000", 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7009999999999999e-05 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:6 3.2 e43/15 3:5cea7035:::rbd_info:head v 43'4, mlcod=42'2)", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:47.249010+0000", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.60218985599999997, 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084398100000000005, 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249010+0000", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:47.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249010+0000", 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249010+0000", 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249017+0000", 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7950000000000002e-06 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249028+0000", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1168e-05 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249035+0000", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2810000000000003e-06 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249091+0000", 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5407e-05 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249116+0000", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5052999999999999e-05 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249809+0000", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00069298999999999999 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:47.249854+0000", 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5479000000000003e-05 2024-04-24T15:59:47.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:47.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:47.861 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:47.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:47.941 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:47.943+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:48.009 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:48.010 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:48.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:48.042 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:48.044+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:48.143 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:48.145+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:48.159 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:48.160 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:48.160 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:48.160 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:48.160 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:48.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:48.244 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:48.247+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:48.346 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:48.347+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4631.0:2 1.6 e42/12 1:774f7988:ns::rbd_directory:head v 42'1, mlcod=0'0)", 2024-04-24T15:59:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:46.763829+0000", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.6424219469999999, 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079992099999999999, 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763829+0000", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763829+0000", 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763827+0000", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763844+0000", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6628000000000001e-05 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763862+0000", 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8022000000000001e-05 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763873+0000", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314e-05 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763915+0000", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2002999999999997e-05 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.763933+0000", 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7255999999999999e-05 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.764592+0000", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065909900000000001 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.764629+0000", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7554e-05 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:0 3.3 e42/15 3:da3fcecf:::testrbdnbd37497.rbd:head v 42'1, mlcod=0'0)", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:46.800135+0000", 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.6061165549999998, 2024-04-24T15:59:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064255400000000004, 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800135+0000", 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800135+0000", 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800134+0000", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800143+0000", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7669999999999992e-06 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800158+0000", 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5089000000000001e-05 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800166+0000", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.644e-06 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800213+0000", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7441999999999997e-05 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800228+0000", 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4905999999999999e-05 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800739+0000", 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00051066499999999999 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800777+0000", 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8389999999999997e-05 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:1 3.3 e42/15 3:c336d817:::rbd_id.testrbdnbd37497:head v 42'2, mlcod=0'0)", 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:46.800410+0000", 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.6058410219999999, 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069930899999999995, 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800410+0000", 2024-04-24T15:59:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800410+0000", 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800409+0000", 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800417+0000", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4749999999999996e-06 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800428+0000", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0648999999999999e-05 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800432+0000", 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5360000000000003e-06 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800462+0000", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0139999999999999e-05 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.800476+0000", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3305000000000001e-05 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.801081+0000", 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060538699999999998 2024-04-24T15:59:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:46.801109+0000", 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8549999999999999e-05 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:10 3.3 e44/15 3:c336d817:::rbd_id.testrbdnbd37497:head v 44'3, mlcod=42'1)", 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.269520+0000", 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.136730879, 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072392300000000001, 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269520+0000", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269520+0000", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269520+0000", 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269527+0000", 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2300000000000002e-06 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269540+0000", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3020000000000001e-05 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269550+0000", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2969999999999992e-06 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269608+0000", 2024-04-24T15:59:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8891000000000002e-05 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.269625+0000", 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6572999999999999e-05 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.270199+0000", 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057374900000000003 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.270244+0000", 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5506000000000001e-05 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:13 1.0 e44/12 1:081f1115:::rbd_object_map.129ad8b6ae3f:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.276569+0000", 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.12968206800000001, 2024-04-24T15:59:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087090900000000001, 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276569+0000", 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276569+0000", 2024-04-24T15:59:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276569+0000", 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967296 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276577+0000", 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2930000000000001e-06 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276589+0000", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1827999999999999e-05 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276607+0000", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8121000000000001e-05 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276655+0000", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8238000000000001e-05 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.276672+0000", 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.613e-05 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.277379+0000", 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070746100000000005 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.277440+0000", 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1050000000000007e-05 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.087491029999999997, 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T15:59:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.086339351999999994, 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T15:59:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T15:59:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.071319568, 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T15:59:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T15:59:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:48.439 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:48.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:48.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:48.446 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:48.448+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:48.519 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-04-24T15:59:48.520 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.P3d5b6KQ4E/data testrbdnbd37497 2024-04-24T15:59:48.547 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:48.549+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:48.648 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:48.649+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:48.666 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:48.666 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:48.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:48.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:48.702 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:48.723 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:48.748 INFO:tasks.workunit.client.0.smithi122.stderr:+ ceph osd pool create new_default_pool37497 8 2024-04-24T15:59:48.749 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:48.749 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:48.750+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:48.769 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:48.788 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:48.849 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:48.849 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:48.850 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:48.851+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:48.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:48.951 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:48.953+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:48.999 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:48.999 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:49.054 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:49.056+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:49.149 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:49.150 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.153 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:49.155+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:49.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:49.254 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:49.256+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:49.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T15:59:49.349 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:49.349 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.355 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:49.357+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:49.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:49.378 INFO:tasks.workunit.client.0.smithi122.stderr:pool 'new_default_pool37497' already exists 2024-04-24T15:59:49.388 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd pool init new_default_pool37497 2024-04-24T15:59:49.457 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:49.459+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:49.542 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:49.542 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:49.597 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:49.599+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:49.659 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:49.661+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:49.681 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:49.681 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:49.775 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:49.777+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:49.855 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:49.855 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:49.861 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:49.863+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:49.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:49.962 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:49.964+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:50.028 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.028 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:50.081 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:50.083+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:50.161 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.161 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.164 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:50.166+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:50.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:50.257 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:50.257 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:50.257 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T15:59:50.257 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T15:59:50.257 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T15:59:50.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T15:59:50.267 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T15:59:50.301 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:50.303+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:50.345 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.345 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:50.388 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:50.388 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T15:59:50.388 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:50.388 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:50.389+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:50.467 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:50.469+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:50.511 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.511 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:50.568 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:50.570+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:50.644 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.644 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:50.697 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:50.699+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:50.770 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:50.772+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:50.776 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.776 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:50.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:50.871 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:50.873+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:50.950 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:50.951 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:50.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:50.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:50.971 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:50.973+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:50.988 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:51.008 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:51.029 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:51.049 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:51.069 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:51.070 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:51.072 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:51.074+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:51.101 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:51.102 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:51.102 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:51.102 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:51.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:51.197 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:51.199+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:51.274 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:51.276+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:51.297 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:51.297 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:51.297 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:51.297 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:51.297 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:51.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:51.393 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:51.395+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:51.476 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:51.478+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:51.493 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:51.494 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:51.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:51.494 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:51.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1754043539999999, 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.495 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T15:59:51.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T15:59:51.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T15:59:51.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T15:59:51.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T15:59:51.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T15:59:51.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T15:59:51.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T15:59:51.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T15:59:51.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1742526760000001, 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.507 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:51.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T15:59:51.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T15:59:51.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T15:59:51.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T15:59:51.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T15:59:51.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T15:59:51.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T15:59:51.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T15:59:51.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T15:59:51.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T15:59:51.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T15:59:51.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T15:59:51.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1592328919999999, 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T15:59:51.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T15:59:51.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.525 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1383005540000002, 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T15:59:51.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T15:59:51.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1376786000000001, 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T15:59:51.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T15:59:51.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T15:59:51.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.538 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:51.538 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T15:59:51.538 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.1096940790000001, 2024-04-24T15:59:51.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T15:59:51.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.539 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.539 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T15:59:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T15:59:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.546 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0963783889999998, 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T15:59:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T15:59:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T15:59:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0836665879999998, 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.552 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T15:59:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:8 3.3 e44/15 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'6, mlcod=44'4)", 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.448407+0000", 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0457574190000001, 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0083961290000000004, 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448404+0000", 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T15:59:51.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452217+0000", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038123480000000001 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452241+0000", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4261e-05 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452253+0000", 2024-04-24T15:59:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2130000000000001e-05 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452299+0000", 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6060999999999998e-05 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452328+0000", 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.915e-05 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456722+0000", 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043940510000000004 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456803+0000", 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0901000000000004e-05 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.561 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0250776830000001, 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.562 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0056422270000001, 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T15:59:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T15:59:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.9931622450000002, 2024-04-24T15:59:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T15:59:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T15:59:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T15:59:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.865727197, 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T15:59:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T15:59:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T15:59:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T15:59:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.86504045, 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8638859760000002, 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T15:59:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T15:59:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8602385510000001, 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T15:59:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T15:59:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T15:59:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8402784599999999, 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T15:59:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T15:59:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T15:59:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T15:59:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T15:59:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8388705509999999, 2024-04-24T15:59:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T15:59:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T15:59:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T15:59:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T15:59:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8382345020000002, 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T15:59:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:49 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.659242+0000", 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.8349226170000001, 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065843639999999997, 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T15:59:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659242+0000", 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659242+0000", 2024-04-24T15:59:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659240+0000", 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659247+0000", 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8589999999999999e-06 2024-04-24T15:59:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659256+0000", 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2069999999999995e-06 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659260+0000", 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2470000000000002e-06 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659313+0000", 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3103999999999998e-05 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659458+0000", 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014483599999999999 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659961+0000", 2024-04-24T15:59:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050371400000000005 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.660914+0000", 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095237800000000004 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665759+0000", 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048452699999999996 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665797+0000", 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8049000000000001e-05 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665826+0000", 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9118000000000001e-05 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:51.634 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:51.636 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:51.638+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:51.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:51.737 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:51.739+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:51.825 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:51.825 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:51.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:51.878 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:51.880+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:51.939 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:51.941+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:52.018 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:52.018 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:52.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:52.074 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:52.076+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:52.141 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:52.143+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.865168068, 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T15:59:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T15:59:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8555480040000001, 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8351474369999998, 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8222156350000001, 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8221818839999999, 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8206594740000002, 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T15:59:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T15:59:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T15:59:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8152483579999998, 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T15:59:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T15:59:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8074998400000002, 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8053698169999999, 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T15:59:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7896668600000001, 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T15:59:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7896388729999999, 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T15:59:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T15:59:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.772867797, 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7675242730000003, 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7553140279999999, 2024-04-24T15:59:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7478788629999999, 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:57 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.424098+0000", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.7469326860000001, 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038713490000000003, 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424097+0000", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424102+0000", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3149999999999999e-06 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424118+0000", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5625999999999999e-05 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424125+0000", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2960000000000001e-06 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424490+0000", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000364781 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424569+0000", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8584000000000005e-05 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424869+0000", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030080800000000001 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430343+0000", 2024-04-24T15:59:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054732189999999997 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462762+0000", 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032419199000000003 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462789+0000", 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7463000000000001e-05 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462811+0000", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2048999999999998e-05 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:59 3.2 3:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.471298+0000", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.6997324909999998, 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039387966000000003, 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 59 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471297+0000", 2024-04-24T15:59:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479102+0000", 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078047949999999998 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479124+0000", 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2449000000000001e-05 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479132+0000", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6190000000000002e-06 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479193+0000", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1668999999999995e-05 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510682+0000", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031488457999999997 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510686+0000", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2150000000000003e-06 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.5636229639999999, 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.559387037, 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.5525472709999999, 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T15:59:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T15:59:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T15:59:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:52.262 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:52.264 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:52.265+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:52.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:52.274 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd-nbd 2024-04-24T15:59:52.274 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd-nbd 2024-04-24T15:59:52.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T15:59:52.292 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: must specify command 2024-04-24T15:59:52.293 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.293 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-04-24T15:59:52.293 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd-nbd INVALIDCMD 2024-04-24T15:59:52.311 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-04-24T15:59:52.311 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.312 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:52.313 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -ne 0 ']' 2024-04-24T15:59:52.313 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd37497 2024-04-24T15:59:52.313 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd map testrbdnbd37497 2024-04-24T15:59:52.364 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:52.366+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:52.404 INFO:tasks.workunit.client.0.smithi122.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-04-24T15:59:52.406 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: failed to find unused device 2024-04-24T15:59:52.411 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-04-24T15:59:52.414 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.414 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T15:59:52.414 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T15:59:52.414 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:52.414 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:52.415 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:52.415 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T15:59:52.417 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T15:59:52.417 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:52.417 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T15:59:52.458 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:52.459 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:52.465 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:52.467+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:52.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:52.496 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-04-24T15:59:52.499 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd37497 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd37497 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:52.500 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd-nbd 2024-04-24T15:59:52.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd-nbd 2024-04-24T15:59:52.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:52.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd37497 2024-04-24T15:59:52.567 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:52.568+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:52.571 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-04-24T15:59:52.582 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.582 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-04-24T15:59:52.582 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format INVALID list 2024-04-24T15:59:52.629 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: the argument for option '--format' is invalid 2024-04-24T15:59:52.644 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.644 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-04-24T15:59:52.651 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:52.652 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:52.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:52.667 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:52.669+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:52.690 INFO:tasks.workunit.client.0.smithi122.stdout:[] 2024-04-24T15:59:52.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format xml list 2024-04-24T15:59:52.738 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T15:59:52.739 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T15:59:52.739 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T15:59:52.740 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T15:59:52.740 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T15:59:52.742 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T15:59:52.742 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T15:59:52.742 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T15:59:52.768 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:52.770+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:52.830 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T15:59:52.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T15:59:52.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T15:59:52.831 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T15:59:52.831 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T15:59:52.831 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T15:59:52.846 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:52.846 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:52.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:52.885 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=37824 2024-04-24T15:59:52.885 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 37824 2024-04-24T15:59:52.885 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 37824 -C rbd-nbd 2024-04-24T15:59:52.894 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T15:59:52.894 INFO:tasks.workunit.client.0.smithi122.stdout: 37824 ? 00:00:00 rbd-nbd 2024-04-24T15:59:52.895 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd37497 2024-04-24T15:59:52.895 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd37497 2024-04-24T15:59:52.895 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:52.895 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:52.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:52.896 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd-nbd 2024-04-24T15:59:52.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd-nbd 2024-04-24T15:59:52.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:52.898 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd37497 2024-04-24T15:59:52.902 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:52.904+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:52.970 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:52.972+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:52.972 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-04-24T15:59:52.979 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:52.979 INFO:tasks.workunit.client.0.smithi122.stderr:+ dev1=/dev/nbd0 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 37824 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=37824 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:52.980 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T15:59:52.982 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T15:59:52.982 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:52.982 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:52.993 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:52.993 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:52.993 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:52.993 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:53.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:53.071 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:53.073+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:53.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:53.166 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:53.167 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:53.167 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:53.167 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:53.167 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:53.172 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:53.173+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:53.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:53.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:53.266 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:53.272 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:53.274+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T15:59:53.290 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:53.310 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:53.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:53.344 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:53.345 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:53.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:53.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:53.400 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:53.403+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:53.474 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:53.476+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:53.480 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:53.480 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:53.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:53.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T15:59:53.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^37824\b' 2024-04-24T15:59:53.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^37824\b' 2024-04-24T15:59:53.575 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:53.577+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:53.601 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:53.601 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T15:59:53.602 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *37824\b' 2024-04-24T15:59:53.602 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *37824\b' 2024-04-24T15:59:53.606 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:53.606 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T15:59:53.654 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:53.654 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:53.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:53.705 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:53.707+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:53.777 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:53.779+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:53.828 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:53.828 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:53.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:53.881 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:53.883+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:53.961 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:53.961 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:53.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:54.011 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:54.014+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:54.080 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:54.081+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:54.108 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T15:59:54.108 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd37497 2024-04-24T15:59:54.108 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T15:59:54.108 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T15:59:54.109 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T15:59:54.110 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd-nbd 2024-04-24T15:59:54.112 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd-nbd 2024-04-24T15:59:54.112 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T15:59:54.112 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd37497 2024-04-24T15:59:54.133 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:54.134 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:54.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:54.184 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:54.187+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:54.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T15:59:54.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-04-24T15:59:54.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T15:59:54.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep testrbdnbd37497 2024-04-24T15:59:54.247 INFO:tasks.workunit.client.0.smithi122.stdout:37933 rbd testrbdnbd37497 - /dev/nbd0 2024-04-24T15:59:54.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T15:59:54.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T15:59:54.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T15:59:54.248 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T15:59:54.249 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T15:59:54.282 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:54.283+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:54.304 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=37933 2024-04-24T15:59:54.304 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 37933 2024-04-24T15:59:54.304 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 37933 -C rbd-nbd 2024-04-24T15:59:54.308 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:54.308 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:54.309 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T15:59:54.309 INFO:tasks.workunit.client.0.smithi122.stdout: 37933 ? 00:00:00 rbd-nbd 2024-04-24T15:59:54.310 INFO:tasks.workunit.client.0.smithi122.stderr:++ dd if=/tmp/tmp.P3d5b6KQ4E/data bs=1M 2024-04-24T15:59:54.310 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-04-24T15:59:54.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:54.383 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:54.385+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:54.422 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:54.422 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:54.422 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.111213 s, 603 MB/s 2024-04-24T15:59:54.423 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-04-24T15:59:54.423 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T15:59:54.424 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-04-24T15:59:54.424 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T15:59:54.424 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T15:59:54.425 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which dd 2024-04-24T15:59:54.426 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/dd 2024-04-24T15:59:54.426 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T15:59:54.426 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-04-24T15:59:54.477 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:54.478 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:54.484 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:54.486+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:54.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:54.585 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:54.587+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:54.637 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:54.637 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:54.637 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.201273 s, 333 MB/s 2024-04-24T15:59:54.638 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' '209b9987dafd2ec8e3d9279112fb8b2a -' = '209b9987dafd2ec8e3d9279112fb8b2a -' ']' 2024-04-24T15:59:54.638 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/dev/urandom of=/tmp/tmp.P3d5b6KQ4E/data bs=1M count=64 2024-04-24T15:59:54.686 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:54.686 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:54.686 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:54.688+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:54.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:54.787 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:54.788+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:54.835 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:54.835 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:54.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:54.887 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:54.890+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:54.988 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:54.990+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:55.030 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:55.030 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:55.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:55.088 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:55.091+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:55.189 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:55.192+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:55.224 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:55.224 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:55.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:55.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T15:59:55.291 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:55.293+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:55.319 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:55.319 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:55.319 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.669956 s, 100 MB/s 2024-04-24T15:59:55.319 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T15:59:55.319 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:55.320 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:55.320 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:55.320 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T15:59:55.322 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T15:59:55.323 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:55.323 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T15:59:55.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T15:59:55.387 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:55.387 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:55.392 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:55.394+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:55.399 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T15:59:55.399 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T15:59:55.400 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T15:59:55.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T15:59:55.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:55.493 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:55.495+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:55.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:55.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:55.594 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:55.596+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:55.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:55.630 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:55.630 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:55.630 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:55.630 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:55.637 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:55.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:55.664 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:55.688 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:55.695 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:55.697+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:55.795 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:55.797+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:55.832 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:55.832 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:55.832 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T15:59:55.832 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T15:59:55.833 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:55.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:55.861 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:55.861 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:55.861 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.527151 s, 127 MB/s 2024-04-24T15:59:55.863 INFO:tasks.workunit.client.0.smithi122.stderr:++ dd if=/tmp/tmp.P3d5b6KQ4E/data bs=1M 2024-04-24T15:59:55.863 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 50, 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c4fc9996-5ff9-4e71-b213-9020967fae1a", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-04-24T15:58:30.292182+0000", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-04-24T15:59:55.039036+0000", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-04-24T15:58:41.011852+0000", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-04-24T15:58:32.987203+0000", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-04-24T15:59:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 4, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 6, 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-04-24T15:59:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "quincy", 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.444445+0000", 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:55.867 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:55.868 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "44", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 3, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 44, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [ 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: 3 2024-04-24T15:59:55.869 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": 3, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": 3, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T15:59:55.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T15:59:55.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.808270+0000", 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:55.872 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 1, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 1, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 1, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 1, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 1, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:55.873 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "14", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-04-24T15:59:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 3, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cache", 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:48.277449+0000", 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8201, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 4, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 4, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 4, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 4, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 4, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "44", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 3, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 44, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": 1, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "writeback", 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 250, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bloom", 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "target_size": 0, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "seed": 0 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 60, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 8, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [ 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 4, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "new_default_pool37497", 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:59:49.072101+0000", 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T15:59:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "48", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 3, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 48, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T15:59:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T15:59:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-04-24T15:59:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "51c39890-e79b-4030-a4a0-8c98ec2018ba", 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 45, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-04-24T15:59:55.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/41951", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/41951", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6819/41951", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/41951", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.892 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "1fb2523f-db36-46d9-80c4-dc08ac530daa", 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 45, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/41952", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/41952", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/41952", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/41952", 2024-04-24T15:59:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7d4dd13f-26d8-4c39-a6a6-020e91634f4e", 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0.308197021484375, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.900 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6814/41953", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6818/41953", 2024-04-24T15:59:55.902 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/41953", 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/41953", 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "03ed1eb6-8fa9-497c-b769-2743acc0e6aa", 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 45, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6801", 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2024-04-24T15:59:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/41923", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6807/41923", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6815/41923", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6811/41923", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f7ff11ec-62d6-44f3-bea1-7ff62d280e2f", 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 45, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2024-04-24T15:59:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2024-04-24T15:59:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6817/41924", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6819/41924", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6823/41924", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6821/41924", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "423ca5ac-f6a3-496e-a4c5-9ac205e2e474", 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 34, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T15:59:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6800", 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T15:59:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T15:59:55.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6802/41925", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6806/41925", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6814/41925", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6810/41925", 2024-04-24T15:59:55.919 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.171683+0000", 2024-04-24T15:59:55.920 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.085846+0000", 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.921 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.118237+0000", 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.172469+0000", 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.139953+0000", 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T15:59:55.923 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.084518+0000", 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": {}, 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-04-24T15:59:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 4, 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [ 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 3, 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-04-24T15:59:55.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T15:59:55.927 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:55.929 INFO:tasks.thrashosds.thrasher:j is {'epoch': 50, 'fsid': 'c4fc9996-5ff9-4e71-b213-9020967fae1a', 'created': '2024-04-24T15:58:30.292182+0000', 'modified': '2024-04-24T15:59:55.039036+0000', 'last_up_change': '2024-04-24T15:58:41.011852+0000', 'last_in_change': '2024-04-24T15:58:32.987203+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-24T15:58:41.444445+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 44, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-24T15:58:41.808270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T15:58:48.277449+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 44, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool37497', 'create_time': '2024-04-24T15:59:49.072101+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '51c39890-e79b-4030-a4a0-8c98ec2018ba', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 41951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 41951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6817', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 41951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6813', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 41951}]}, 'public_addr': '172.21.15.32:6809/41951', 'cluster_addr': '172.21.15.32:6811/41951', 'heartbeat_back_addr': '172.21.15.32:6819/41951', 'heartbeat_front_addr': '172.21.15.32:6815/41951', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fb2523f-db36-46d9-80c4-dc08ac530daa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 41952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 41952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 41952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 41952}]}, 'public_addr': '172.21.15.32:6801/41952', 'cluster_addr': '172.21.15.32:6803/41952', 'heartbeat_back_addr': '172.21.15.32:6807/41952', 'heartbeat_front_addr': '172.21.15.32:6805/41952', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7d4dd13f-26d8-4c39-a6a6-020e91634f4e', 'up': 1, 'in': 1, 'weight': 0.308197021484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6814', 'nonce': 41953}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6818', 'nonce': 41953}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 41953}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 41953}]}, 'public_addr': '172.21.15.32:6814/41953', 'cluster_addr': '172.21.15.32:6818/41953', 'heartbeat_back_addr': '172.21.15.32:6823/41953', 'heartbeat_front_addr': '172.21.15.32:6821/41953', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '03ed1eb6-8fa9-497c-b769-2743acc0e6aa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6803', 'nonce': 41923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6807', 'nonce': 41923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6815', 'nonce': 41923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6811', 'nonce': 41923}]}, 'public_addr': '172.21.15.86:6803/41923', 'cluster_addr': '172.21.15.86:6807/41923', 'heartbeat_back_addr': '172.21.15.86:6815/41923', 'heartbeat_front_addr': '172.21.15.86:6811/41923', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7ff11ec-62d6-44f3-bea1-7ff62d280e2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6816', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6817', 'nonce': 41924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6818', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6819', 'nonce': 41924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6822', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6823', 'nonce': 41924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6820', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6821', 'nonce': 41924}]}, 'public_addr': '172.21.15.86:6817/41924', 'cluster_addr': '172.21.15.86:6819/41924', 'heartbeat_back_addr': '172.21.15.86:6823/41924', 'heartbeat_front_addr': '172.21.15.86:6821/41924', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '423ca5ac-f6a3-496e-a4c5-9ac205e2e474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6802', 'nonce': 41925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6806', 'nonce': 41925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6814', 'nonce': 41925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6810', 'nonce': 41925}]}, 'public_addr': '172.21.15.86:6802/41925', 'cluster_addr': '172.21.15.86:6806/41925', 'heartbeat_back_addr': '172.21.15.86:6814/41925', 'heartbeat_front_addr': '172.21.15.86:6810/41925', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.171683+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.085846+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.118237+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.172469+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.139953+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.084518+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 1, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T15:59:55.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T15:59:55.930 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:55.931+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:55.966 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T15:59:55.966 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T15:59:55.966 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.102584 s, 654 MB/s 2024-04-24T15:59:55.967 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --no-progress export testrbdnbd37497 - 2024-04-24T15:59:55.967 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-04-24T15:59:56.030 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:56.032+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:56.031 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T15:59:56.032+0000 7f88f6e05700 0 -- 172.21.15.122:0/205328714 >> [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] conn(0x7f88d0008af0 msgr2=0x7f88d000af70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T15:59:56.039 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T15:59:56.041+0000 7f88effff700 0 -- 172.21.15.122:0/205328714 >> [v2:172.21.15.86:6816/41924,v1:172.21.15.86:6817/41924] conn(0x556e8d042300 msgr2=0x556e8d0446e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T15:59:56.086 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:56.086 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T15:59:56.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T15:59:56.086 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T15:59:56.086 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:56.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:56.141 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:56.143+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' '84e5955d9429e2114541ea510cbb75ee -' = '84e5955d9429e2114541ea510cbb75ee -' ']' 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 37933 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=37933 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:56.159 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:56.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:56.160 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T15:59:56.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T15:59:56.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:56.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:56.225 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.225 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:56.225 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:56.225 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:56.231 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:56.233+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:56.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T15:59:56.340 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T15:59:56.340 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:56.341+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:56.342 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T15:59:56.355 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-24T15:59:56.286390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189334258,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":3,"num_read":416,"num_read_kb":323,"num_write":276,"num_write_kb":133565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":2161240,"kb_used_data":372984,"kb_used_omap":9,"kb_used_meta":1788150,"kb_avail":564069800,"statfs":{"total":579820584960,"available":577607475200,"internally_reserved":0,"allocated":381935616,"data_stored":379901930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1831065616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":188743890,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":-1,"num_read":296,"num_read_kb":222,"num_write":115,"num_write_kb":131147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.522126"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.263909+0000","last_change":"2024-04-24T15:59:48.269151+0000","last_active":"2024-04-24T15:59:49.263909+0000","last_peered":"2024-04-24T15:59:49.263909+0000","last_clean":"2024-04-24T15:59:49.263909+0000","last_became_active":"2024-04-24T15:59:28.190640+0000","last_became_peered":"2024-04-24T15:59:28.190640+0000","last_unstale":"2024-04-24T15:59:49.263909+0000","last_undegraded":"2024-04-24T15:59:49.263909+0000","last_fullsized":"2024-04-24T15:59:49.263909+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:27:16.674670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.270475+0000","last_change":"2024-04-24T15:59:50.269654+0000","last_active":"2024-04-24T15:59:50.270475+0000","last_peered":"2024-04-24T15:59:50.270475+0000","last_clean":"2024-04-24T15:59:50.270475+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T15:59:50.270475+0000","last_undegraded":"2024-04-24T15:59:50.270475+0000","last_fullsized":"2024-04-24T15:59:50.270475+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.262782+0000","last_change":"2024-04-24T15:59:48.255715+0000","last_active":"2024-04-24T15:59:50.262782+0000","last_peered":"2024-04-24T15:59:50.262782+0000","last_clean":"2024-04-24T15:59:50.262782+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T15:59:50.262782+0000","last_undegraded":"2024-04-24T15:59:50.262782+0000","last_fullsized":"2024-04-24T15:59:50.262782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.311840+0000","last_change":"2024-04-24T15:59:50.311840+0000","last_active":"2024-04-24T15:59:50.311840+0000","last_peered":"2024-04-24T15:59:50.311840+0000","last_clean":"2024-04-24T15:59:50.311840+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T15:59:50.311840+0000","last_undegraded":"2024-04-24T15:59:50.311840+0000","last_fullsized":"2024-04-24T15:59:50.311840+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.267033+0000","last_change":"2024-04-24T15:59:50.266452+0000","last_active":"2024-04-24T15:59:50.267033+0000","last_peered":"2024-04-24T15:59:50.267033+0000","last_clean":"2024-04-24T15:59:50.267033+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T15:59:50.267033+0000","last_undegraded":"2024-04-24T15:59:50.267033+0000","last_fullsized":"2024-04-24T15:59:50.267033+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":65,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.262720+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T15:59:50.262720+0000","last_peered":"2024-04-24T15:59:50.262720+0000","last_clean":"2024-04-24T15:59:50.262720+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T15:59:50.262720+0000","last_undegraded":"2024-04-24T15:59:50.262720+0000","last_fullsized":"2024-04-24T15:59:50.262720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":22,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"13'117","reported_seq":191,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.262795+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T15:59:49.262795+0000","last_peered":"2024-04-24T15:59:49.262795+0000","last_clean":"2024-04-24T15:59:49.262795+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T15:59:49.262795+0000","last_undegraded":"2024-04-24T15:59:49.262795+0000","last_fullsized":"2024-04-24T15:59:49.262795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'23","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.262976+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T15:59:50.262976+0000","last_peered":"2024-04-24T15:59:50.262976+0000","last_clean":"2024-04-24T15:59:50.262976+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T15:59:50.262976+0000","last_undegraded":"2024-04-24T15:59:50.262976+0000","last_fullsized":"2024-04-24T15:59:50.262976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":100,"num_read_kb":82,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.312498+0000","last_change":"2024-04-24T15:59:50.311357+0000","last_active":"2024-04-24T15:59:50.312498+0000","last_peered":"2024-04-24T15:59:50.312498+0000","last_clean":"2024-04-24T15:59:50.312498+0000","last_became_active":"2024-04-24T15:59:50.310981+0000","last_became_peered":"2024-04-24T15:59:50.310981+0000","last_unstale":"2024-04-24T15:59:50.312498+0000","last_undegraded":"2024-04-24T15:59:50.312498+0000","last_fullsized":"2024-04-24T15:59:50.312498+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:30:51.187957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'7","reported_seq":58,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.263815+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T15:59:49.263815+0000","last_peered":"2024-04-24T15:59:49.263815+0000","last_clean":"2024-04-24T15:59:49.263815+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T15:59:49.263815+0000","last_undegraded":"2024-04-24T15:59:49.263815+0000","last_fullsized":"2024-04-24T15:59:49.263815+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.0","version":"44'30","reported_seq":97,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.262645+0000","last_change":"2024-04-24T15:59:48.256825+0000","last_active":"2024-04-24T15:59:50.262645+0000","last_peered":"2024-04-24T15:59:50.262645+0000","last_clean":"2024-04-24T15:59:50.262645+0000","last_became_active":"2024-04-24T15:59:28.190693+0000","last_became_peered":"2024-04-24T15:59:28.190693+0000","last_unstale":"2024-04-24T15:59:50.262645+0000","last_undegraded":"2024-04-24T15:59:50.262645+0000","last_fullsized":"2024-04-24T15:59:50.262645+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:14.816623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":12582950,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":104,"num_write":36,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.312024+0000","last_change":"2024-04-24T15:59:50.312024+0000","last_active":"2024-04-24T15:59:50.312024+0000","last_peered":"2024-04-24T15:59:50.312024+0000","last_clean":"2024-04-24T15:59:50.312024+0000","last_became_active":"2024-04-24T15:59:50.311714+0000","last_became_peered":"2024-04-24T15:59:50.311714+0000","last_unstale":"2024-04-24T15:59:50.312024+0000","last_undegraded":"2024-04-24T15:59:50.312024+0000","last_fullsized":"2024-04-24T15:59:50.312024+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:57.999783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":56,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.262870+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T15:59:49.262870+0000","last_peered":"2024-04-24T15:59:49.262870+0000","last_clean":"2024-04-24T15:59:49.262870+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T15:59:49.262870+0000","last_undegraded":"2024-04-24T15:59:49.262870+0000","last_fullsized":"2024-04-24T15:59:49.262870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.2","version":"44'10","reported_seq":59,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.263108+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T15:59:50.263108+0000","last_peered":"2024-04-24T15:59:50.263108+0000","last_clean":"2024-04-24T15:59:50.263108+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T15:59:50.263108+0000","last_undegraded":"2024-04-24T15:59:50.263108+0000","last_fullsized":"2024-04-24T15:59:50.263108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:45:33.897580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":16777235,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":8,"num_read_kb":8,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.270472+0000","last_change":"2024-04-24T15:59:50.269637+0000","last_active":"2024-04-24T15:59:50.270472+0000","last_peered":"2024-04-24T15:59:50.270472+0000","last_clean":"2024-04-24T15:59:50.270472+0000","last_became_active":"2024-04-24T15:59:50.269342+0000","last_became_peered":"2024-04-24T15:59:50.269342+0000","last_unstale":"2024-04-24T15:59:50.270472+0000","last_undegraded":"2024-04-24T15:59:50.270472+0000","last_fullsized":"2024-04-24T15:59:50.270472+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:22:26.944617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.309900+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T15:59:50.309900+0000","last_peered":"2024-04-24T15:59:50.309900+0000","last_clean":"2024-04-24T15:59:50.309900+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T15:59:50.309900+0000","last_undegraded":"2024-04-24T15:59:50.309900+0000","last_fullsized":"2024-04-24T15:59:50.309900+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"44'11","reported_seq":60,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.263833+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T15:59:49.263833+0000","last_peered":"2024-04-24T15:59:49.263833+0000","last_clean":"2024-04-24T15:59:49.263833+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T15:59:49.263833+0000","last_undegraded":"2024-04-24T15:59:49.263833+0000","last_fullsized":"2024-04-24T15:59:49.263833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:20.514114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":12582928,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":8,"num_read_kb":2,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.312887+0000","last_change":"2024-04-24T15:59:50.311959+0000","last_active":"2024-04-24T15:59:50.312887+0000","last_peered":"2024-04-24T15:59:50.312887+0000","last_clean":"2024-04-24T15:59:50.312887+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T15:59:50.312887+0000","last_undegraded":"2024-04-24T15:59:50.312887+0000","last_fullsized":"2024-04-24T15:59:50.312887+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"44'3","reported_seq":52,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-24T15:59:49.263005+0000","last_change":"2024-04-24T15:59:48.261056+0000","last_active":"2024-04-24T15:59:49.263005+0000","last_peered":"2024-04-24T15:59:49.263005+0000","last_clean":"2024-04-24T15:59:49.263005+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T15:59:49.263005+0000","last_undegraded":"2024-04-24T15:59:49.263005+0000","last_fullsized":"2024-04-24T15:59:49.263005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"44'7","reported_seq":66,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.309892+0000","last_change":"2024-04-24T15:59:48.260366+0000","last_active":"2024-04-24T15:59:50.309892+0000","last_peered":"2024-04-24T15:59:50.309892+0000","last_clean":"2024-04-24T15:59:50.309892+0000","last_became_active":"2024-04-24T15:59:28.271464+0000","last_became_peered":"2024-04-24T15:59:28.271464+0000","last_unstale":"2024-04-24T15:59:50.309892+0000","last_undegraded":"2024-04-24T15:59:50.309892+0000","last_fullsized":"2024-04-24T15:59:50.309892+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:41.867779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165839,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":7,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.2","version":"47'1","reported_seq":10,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-24T15:59:51.266727+0000","last_change":"2024-04-24T15:59:50.311533+0000","last_active":"2024-04-24T15:59:51.266727+0000","last_peered":"2024-04-24T15:59:51.266727+0000","last_clean":"2024-04-24T15:59:51.266727+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T15:59:51.266727+0000","last_undegraded":"2024-04-24T15:59:51.266727+0000","last_fullsized":"2024-04-24T15:59:51.266727+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":54526063,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":243,"num_read_kb":196,"num_write":76,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":25,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109084672,"data_stored":109052126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217819,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":57,"num_read_kb":27,"num_write":40,"num_write_kb":131080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268435638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367540,"kb_used_data":70176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94004300,"statfs":{"total":96636764160,"available":96260403200,"internally_reserved":0,"allocated":71860224,"data_stored":71512625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.434}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.29699999999999999}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32500000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375716,"kb_used_data":78356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93996124,"statfs":{"total":96636764160,"available":96252030976,"internally_reserved":0,"allocated":80236544,"data_stored":79895571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.26000000000000001}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.216}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.309},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.34699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372184,"kb_used_data":74832,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999656,"statfs":{"total":96636764160,"available":96255647744,"internally_reserved":0,"allocated":76627968,"data_stored":76291558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.436},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45800000000000002}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.36399999999999999}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.501},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45000000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.377},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39400000000000002}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.34499999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297876,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073964,"statfs":{"total":96636764160,"available":96331739136,"internally_reserved":0,"allocated":528384,"data_stored":203665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.315},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.33100000000000002}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58199999999999996}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.50800000000000001}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53700000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640274,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375712,"kb_used_data":78352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93996128,"statfs":{"total":96636764160,"available":96252035072,"internally_reserved":0,"allocated":80232448,"data_stored":79895533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35599999999999998}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.33700000000000002}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42299999999999999}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.437},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.45900000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640275,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372212,"kb_used_data":70752,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93999628,"statfs":{"total":96636764160,"available":96255619072,"internally_reserved":0,"allocated":72450048,"data_stored":72102978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.309},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.36199999999999999}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47399999999999998}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.436},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51400000000000001}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54530048,"data_stored":54525990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75505664,"data_stored":75497525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54534144,"data_stored":54526005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29368320,"data_stored":29360185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":37761024,"data_stored":37748828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16781312,"data_stored":16777235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:59:56.357 INFO:tasks.thrashosds.thrasher:Setting 4.7 to [3, 4] 2024-04-24T15:59:56.357 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.7', '3', '4'] 2024-04-24T15:59:56.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.7 3 4 2024-04-24T15:59:56.359 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:56.360 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:56.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:56.439 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:56.441+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:56.496 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.496 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:56.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1867519879999993, 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T15:59:56.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1787859580000006, 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T15:59:56.502 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T15:59:56.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T15:59:56.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T15:59:56.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1787517629999993, 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T15:59:56.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T15:59:56.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1787331250000008, 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T15:59:56.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T15:59:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T15:59:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T15:59:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T15:59:56.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.171410388, 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T15:59:56.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T15:59:56.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T15:59:56.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1497513650000002, 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T15:59:56.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.526 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1455027310000006, 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T15:59:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T15:59:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1454682130000009, 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.1153071089999997, 2024-04-24T15:59:56.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T15:59:56.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.0642121699999993, 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T15:59:56.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T15:59:56.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T15:59:56.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T15:59:56.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.0106936579999992, 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.543 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8873534979999995, 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8704840430000003, 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8622676179999997, 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T15:59:56.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T15:59:56.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T15:59:56.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.84872747, 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T15:59:56.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T15:59:56.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T15:59:56.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T15:59:56.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8468428010000002, 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T15:59:56.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T15:59:56.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8262345419999999, 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.816313042, 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T15:59:56.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.577 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8116647839999995, 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T15:59:56.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T15:59:56.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T15:59:56.581 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 7.8101709929999998, 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.585 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:56.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:56.587 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:56.588+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:56.688 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:56.690+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:56.711 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:56.711 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:56.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T15:59:56.738 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T15:59:56.738 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^37933\b' 2024-04-24T15:59:56.738 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^37933\b' 2024-04-24T15:59:56.781 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:56.781 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T15:59:56.781 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *37933\b' 2024-04-24T15:59:56.782 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *37933\b' 2024-04-24T15:59:56.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:56.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T15:59:56.789 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:56.792+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:56.845 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.845 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:56.845 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:56.845 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:56.845 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:56.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T15:59:56.898 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:56.901+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:56.981 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.6750062910000008, 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T15:59:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T15:59:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T15:59:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T15:59:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T15:59:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T15:59:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.6738595049999994, 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T15:59:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T15:59:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T15:59:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.6580073829999993, 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T15:59:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T15:59:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T15:59:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.6352427519999999, 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T15:59:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.6251243960000004, 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T15:59:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5971812809999992, 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T15:59:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5971634550000005, 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T15:59:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T15:59:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5875460570000008, 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5838171459999995, 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.557933083, 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5017114429999996, 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T15:59:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.5016903329999991, 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T15:59:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T15:59:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3888154880000005, 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T15:59:57.060 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T15:59:57.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T15:59:57.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T15:59:57.064 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3793977179999999, 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.065 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T15:59:57.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T15:59:57.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3628815519999993, 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.069 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T15:59:57.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T15:59:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T15:59:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3623251360000008, 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.073 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T15:59:57.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T15:59:57.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3130947319999997, 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T15:59:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T15:59:57.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T15:59:57.085 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.301416905, 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T15:59:57.086 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T15:59:57.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T15:59:57.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T15:59:57.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.3000691460000002, 2024-04-24T15:59:57.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T15:59:57.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.2919415660000002, 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.097 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T15:59:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T15:59:57.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T15:59:57.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T15:59:57.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.104 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:57.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T15:59:57.106 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:57.108+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:57.206 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:57.209+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:57.270 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:57.271 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:57.271 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T15:59:57.271 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:57.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T15:59:57.287 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd37497 2024-04-24T15:59:57.288 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T15:59:57.288 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T15:59:57.289 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T15:59:57.289 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T15:59:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:set 4.7 pg_upmap mapping to [3,4] 2024-04-24T15:59:57.291 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T15:59:57.291 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T15:59:57.291 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd37497 2024-04-24T15:59:57.321 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:57.324+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T15:59:57.393 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T15:59:57.408 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:57.411+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:57.448 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:57.448 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T15:59:57.448 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T15:59:57.448 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T15:59:57.448 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:57.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38058 2024-04-24T15:59:57.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38058 2024-04-24T15:59:57.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38058 -C rbd-nbd 2024-04-24T15:59:57.455 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T15:59:57.456 INFO:tasks.workunit.client.0.smithi122.stdout: 38058 ? 00:00:00 rbd-nbd 2024-04-24T15:59:57.456 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T15:59:57.456 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T15:59:57.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T15:59:57.499 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-04-24T15:59:57.499 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T15:59:57.499 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T15:59:57.510 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:57.512+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:57.611 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:57.613+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.623 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.3005868159999991, 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T15:59:57.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T15:59:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T15:59:57.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2987822359999992, 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T15:59:57.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T15:59:57.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T15:59:57.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.632 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2987508680000008, 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2750271420000008, 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2471694039999992, 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2425398990000005, 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2357323559999998, 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2051921290000003, 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T15:59:57.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.1895524490000007, 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.1503771999999994, 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.1377590800000004, 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T15:59:57.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.0153629429999995, 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T15:59:57.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T15:59:57.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T15:59:57.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.0018337349999999, 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T15:59:57.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T15:59:57.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T15:59:57.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.677 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.9910994070000001, 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T15:59:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.9768032640000008, 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.9445761719999997, 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T15:59:57.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T15:59:57.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.9384803309999992, 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T15:59:57.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T15:59:57.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T15:59:57.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T15:59:57.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.9350257739999996, 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T15:59:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T15:59:57.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 8.8995816530000003, 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.8481218880000001, 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T15:59:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T15:59:57.710 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T15:59:57.710 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T15:59:57.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T15:59:57.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T15:59:57.712 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:57.715+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard /dev/nbd0 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:57.744 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:57.745 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:57.745 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T15:59:57.746 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T15:59:57.746 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:57.746 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-04-24T15:59:57.765 INFO:tasks.workunit.client.0.smithi122.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-04-24T15:59:57.766 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T15:59:57.766 INFO:tasks.workunit.client.0.smithi122.stderr:+ sync 2024-04-24T15:59:57.813 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:57.815+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:57.869 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T15:59:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T15:59:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T15:59:57.869 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T15:59:57.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T15:59:57.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T15:59:57.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T15:59:57.914 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:57.916+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:57.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T15:59:57.951 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T15:59:57.972 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T15:59:57.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T15:59:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T15:59:58.014 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:58.017+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:58.032 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.032 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T15:59:58.128 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:58.130+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:58.216 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:58.218+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:58.225 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.226 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T15:59:58.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T15:59:58.316 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T15:59:58.319+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T15:59:58.418 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:58.420+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:58.422 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.423 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T15:59:58.518 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:58.521+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.614 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.619 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:58.621+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:58.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T15:59:58.720 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:58.723+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:58.806 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.806 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T15:59:58.820 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:58.823+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:58.921 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:58.923+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:58.955 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:58.956 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:58.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T15:59:59.051 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:59.053+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:59.123 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:59.125+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:59.149 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.149 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T15:59:59.244 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:59.246+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:59.302 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T15:59:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.345 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.345 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:59.347+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:59.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T15:59:59.426 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:59.429+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:59.492 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.492 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T15:59:59.527 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T15:59:59.529+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T15:59:59.628 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.628 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.628 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T15:59:59.630+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T15:59:59.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T15:59:59.729 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T15:59:59.731+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T15:59:59.761 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T15:59:59.761 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T15:59:59.802 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.803 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.807 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-04-24T15:59:59.807 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T15:59:59.807 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T15:59:59.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T15:59:59.830 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T15:59:59.832+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T15:59:59.852 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 38058 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38058 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T15:59:59.853 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T15:59:59.854 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T15:59:59.855 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T15:59:59.855 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T15:59:59.855 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T15:59:59.931 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T15:59:59.934+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T15:59:59.937 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T15:59:59.937 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T15:59:59.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:00.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:00.033 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:00.035+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:00.113 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:00.113 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:00.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:00.133 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:00.135+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:00.177 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:00.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:00.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:00.234 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:00.237+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:00.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:00.274 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:00.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:00.294 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.989620134999999, 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:00.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:00.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.988473349, 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.302 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:00.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:00.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.972621226999999, 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.307 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.949856596, 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:00.311 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.939738240000001, 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:00.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:00.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.911795124999999, 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:00.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:00.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.911777299000001, 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:00.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:00.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:00.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.902159900999999, 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:00.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.333 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.89843099, 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.872546927, 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.816325287, 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:00.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:00.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.816304176999999, 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:00.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:00.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:00.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.353 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.703429332000001, 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.694011562, 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.677495395999999, 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.676938979999999, 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.627708576, 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.616030749, 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.61468299, 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.388 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 11.60655541, 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.393 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:00.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:00.396 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:00.397+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:00.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:00.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38058\b' 2024-04-24T16:00:00.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38058\b' 2024-04-24T16:00:00.475 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:00.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:00.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38058\b' 2024-04-24T16:00:00.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38058\b' 2024-04-24T16:00:00.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:00.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:00.496 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:00.498+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:00.561 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:00.561 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:00.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:00.596 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:00.599+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:00.697 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:00.700+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:00.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:00.799 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:00.801+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:00.871 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.548469820999999, 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.546665240999999, 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:00.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.546633872999999, 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:00.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.522910146999999, 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.885 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.495052408999999, 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:00.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.490422904000001, 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.483615361, 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.453075134000001, 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.437435453999999, 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:00.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:00.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:00.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.908 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.398260205, 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:00.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.385642085000001, 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.263245948, 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:00.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.921 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.24971674, 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:00.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:00.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.925 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.238982412, 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.224686268999999, 2024-04-24T16:00:00.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:00.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:00.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:00.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.192459177, 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:00.935 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:00.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:00.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.186363335999999, 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.940 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.182908779, 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.945 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.147464658000001, 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.0960048929999999, 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:00.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:00.958 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:00.961+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:00.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:00.983 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:00.983 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:00.983 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:00.984 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:00.984 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:00.986 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:00.986 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:00.986 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:01.059 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:01.061+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:01.087 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:01.088 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:01.088 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:01.088 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:01.088 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:01.088 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:01.113 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:01.114 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:01.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:01.142 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38231 2024-04-24T16:00:01.143 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38231 2024-04-24T16:00:01.143 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38231 -C rbd-nbd 2024-04-24T16:00:01.149 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:01.149 INFO:tasks.workunit.client.0.smithi122.stdout: 38231 ? 00:00:00 rbd-nbd 2024-04-24T16:00:01.149 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T16:00:01.149 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T16:00:01.166 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:01.168+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:01.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-04-24T16:00:01.192 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T16:00:01.192 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard /dev/nbd0 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:01.235 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:01.236 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:01.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:01.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:01.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-04-24T16:00:01.261 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:01.264+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:01.294 INFO:tasks.workunit.client.0.smithi122.stderr:+ sync 2024-04-24T16:00:01.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:00:01.487 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:01.488 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:01.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:01.489 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T16:00:01.489 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T16:00:01.489 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd37497 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=0 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -lt 67108864 ']' 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 38231 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38231 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:01.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:01.491 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:01.489+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:01.587 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:01.589+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.361088240999999, 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.359936563, 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.344916779, 2024-04-24T16:00:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.323984441, 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.323362487000001, 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.295377966, 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.282062276, 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:01.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:01.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:01.707 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.269350475, 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:01.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:8 3.3 e44/15 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.231441306000001, 2024-04-24T16:00:01.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0083961290000000004, 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448404+0000", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452217+0000", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038123480000000001 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452241+0000", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4261e-05 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452253+0000", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2130000000000001e-05 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452299+0000", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6060999999999998e-05 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452328+0000", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.915e-05 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456722+0000", 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043940510000000004 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456803+0000", 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0901000000000004e-05 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.210761570000001, 2024-04-24T16:00:01.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:01.719 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.191326114000001, 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.178846132, 2024-04-24T16:00:01.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.727 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.051411084, 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:01.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:01.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:01.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.050724337, 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.049569863, 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.738 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.045922438, 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.025962347, 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:01.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.024554437999999, 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.751 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:01.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.023918389, 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.755 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:01.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:01.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:01.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:01.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:01.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:01.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:01.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:01.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.8400628789999995, 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:01.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:01.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:01.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:01.767 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:01.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:01.769 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:01.771+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:01.869 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:01.871+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:01.947 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:01.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:01.948 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:01.948 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:01.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:01.969 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:01.972+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:02.031 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:02.031 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38231\b' 2024-04-24T16:00:02.031 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38231\b' 2024-04-24T16:00:02.070 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:02.072+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:02.072 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:02.072 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:02.073 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38231\b' 2024-04-24T16:00:02.073 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38231\b' 2024-04-24T16:00:02.078 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:02.078 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:02.140 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:02.141 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:02.141 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:02.141 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:02.141 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:02.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:02.171 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:02.174+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:02.272 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:02.275+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:02.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:02.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:02.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:02.307 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T16:00:02.307 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:00:02.307 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:00:02.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T16:00:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.042401288000001, 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.032781224000001, 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.012380657, 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.999448855000001, 2024-04-24T16:00:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.999415104000001, 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.997892694000001, 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.992481578, 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.378 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.98473306, 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.982603037000001, 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.96690008, 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.966872092999999, 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.950101017, 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.944757493000001, 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.932547248000001, 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.925112083, 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:57 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.924165906000001, 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038713490000000003, 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424097+0000", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424102+0000", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3149999999999999e-06 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424118+0000", 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5625999999999999e-05 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424125+0000", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2960000000000001e-06 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424490+0000", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000364781 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424569+0000", 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8584000000000005e-05 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424869+0000", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030080800000000001 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430343+0000", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054732189999999997 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462762+0000", 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032419199000000003 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462789+0000", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7463000000000001e-05 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462811+0000", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2048999999999998e-05 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:59 3.2 3:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.876965711, 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039387966000000003, 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 59 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471297+0000", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479102+0000", 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078047949999999998 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479124+0000", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2449000000000001e-05 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479132+0000", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6190000000000002e-06 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479193+0000", 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1668999999999995e-05 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510682+0000", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031488457999999997 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510686+0000", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2150000000000003e-06 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.740856184, 2024-04-24T16:00:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.736620257, 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.729780491, 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:02.440 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:02.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:02.442 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:02.443+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:02.443 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T16:00:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:02.447 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:02.467 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:02.487 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:02.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:02.527 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:02.542 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:02.544+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:02.547 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:02.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:02.579 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd37497 2024-04-24T16:00:02.580 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:02.580 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:02.581 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:02.581 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:02.583 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:02.583 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:02.583 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd37497 2024-04-24T16:00:02.587 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:02.587 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:02.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:02.684 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:02.686+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:02.693 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:02.744 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:02.745+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:02.748 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38400 2024-04-24T16:00:02.748 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38400 2024-04-24T16:00:02.748 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38400 -C rbd-nbd 2024-04-24T16:00:02.754 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:02.754 INFO:tasks.workunit.client.0.smithi122.stdout: 38400 ? 00:00:00 rbd-nbd 2024-04-24T16:00:02.754 INFO:tasks.workunit.client.0.smithi122.stderr:++ basename /dev/nbd0 2024-04-24T16:00:02.755 INFO:tasks.workunit.client.0.smithi122.stderr:+ devname=nbd0 2024-04-24T16:00:02.755 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T16:00:02.756 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks=65536 2024-04-24T16:00:02.756 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 65536 2024-04-24T16:00:02.756 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd resize rbd/testrbdnbd37497 --size 128M 2024-04-24T16:00:02.782 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:02.782 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:02.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:02.798 INFO:tasks.workunit.client.0.smithi122.stderr: Resizing image: 100% complete...done. 2024-04-24T16:00:02.804 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd info rbd/testrbdnbd37497 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout:rbd image 'testrbdnbd37497': 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: size 128 MiB in 32 objects 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: order 22 (4 MiB objects) 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: snapshot_count: 0 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: id: 129ad8b6ae3f 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: block_name_prefix: rbd_data.129ad8b6ae3f 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: format: 2 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: op_features: 2024-04-24T16:00:02.838 INFO:tasks.workunit.client.0.smithi122.stdout: flags: 2024-04-24T16:00:02.839 INFO:tasks.workunit.client.0.smithi122.stdout: create_timestamp: Wed Apr 24 15:59:48 2024 2024-04-24T16:00:02.839 INFO:tasks.workunit.client.0.smithi122.stdout: access_timestamp: Wed Apr 24 15:59:48 2024 2024-04-24T16:00:02.839 INFO:tasks.workunit.client.0.smithi122.stdout: modify_timestamp: Wed Apr 24 15:59:48 2024 2024-04-24T16:00:02.840 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T16:00:02.842 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks2=131072 2024-04-24T16:00:02.842 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 131072 2024-04-24T16:00:02.842 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 131072 -eq 131072 2024-04-24T16:00:02.842 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd resize rbd/testrbdnbd37497 --allow-shrink --size 64M 2024-04-24T16:00:02.880 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:02.882+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:02.886 INFO:tasks.workunit.client.0.smithi122.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-04-24T16:00:02.894 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks2=65536 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 65536 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 65536 -eq 65536 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 38400 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38400 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:02.896 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:02.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:02.897 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:02.898 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:02.898 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:02.898 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:02.946 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:02.948+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:02.977 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:02.977 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38400\b' 2024-04-24T16:00:02.978 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38400\b' 2024-04-24T16:00:02.978 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:02.978 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:02.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:03.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:03.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:03.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38400\b' 2024-04-24T16:00:03.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38400\b' 2024-04-24T16:00:03.023 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:03.023 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:03.047 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:03.049+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:03.165 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:03.168+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:03.250 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:03.252+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:03.351 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:03.354+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:03.476 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:03.479+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:03.525 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd37497 2024-04-24T16:00:03.525 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:03.525 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:03.526 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:03.526 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:03.528 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:03.528 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:03.528 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd37497 2024-04-24T16:00:03.553 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:03.556+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:03.632 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:03.632 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd list 2024-04-24T16:00:03.633 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v pool=rbd -v img=testrbdnbd37497 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-04-24T16:00:03.654 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:03.656+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:03.685 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38539 2024-04-24T16:00:03.685 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38539 2024-04-24T16:00:03.685 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38539 -C rbd-nbd 2024-04-24T16:00:03.690 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:03.690 INFO:tasks.workunit.client.0.smithi122.stdout: 38539 ? 00:00:00 rbd-nbd 2024-04-24T16:00:03.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-04-24T16:00:03.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:03.691 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:03.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:03.692 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:03.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:03.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:03.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-04-24T16:00:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:03.743 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T16:00:03.743 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T16:00:03.743 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.0397681 s, 1.7 GB/s 2024-04-24T16:00:03.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T16:00:03.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T16:00:03.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:03.745 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:03.745 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:03.745 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:03.747 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:03.747 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:03.747 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T16:00:03.757 INFO:tasks.workunit.client.0.smithi122.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-04-24T16:00:03.757 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:03.758 INFO:tasks.workunit.client.0.smithi122.stderr:0+0 records out 2024-04-24T16:00:03.758 INFO:tasks.workunit.client.0.smithi122.stderr:0 bytes copied, 0.000447945 s, 0.0 kB/s 2024-04-24T16:00:03.762 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:03.762 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 38539 2024-04-24T16:00:03.762 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38539 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:03.763 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:03.764 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:03.764 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:03.765 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:03.783 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:03.785+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:03.856 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:03.859+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:03.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:03.960 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:03.963+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:04.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:04.091 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:04.094+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:04.160 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:04.162+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:04.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:04.265 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:04.268+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:04.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:00:04.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:04.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38539\b' 2024-04-24T16:00:04.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38539\b' 2024-04-24T16:00:04.362 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:04.365+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:04.382 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:04.383 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:04.383 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38539\b' 2024-04-24T16:00:04.383 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38539\b' 2024-04-24T16:00:04.388 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:04.388 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:04.389 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:04.389 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:04.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:04.464 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:04.466+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:04.542 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:04.542 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:04.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:04.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:04.595 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:04.597+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:04.666 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:04.668+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:04.740 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:04.740 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:04.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:04.753 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:04.773 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:04.773 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:04.775+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:04.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:04.813 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:04.833 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:04.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:04.867 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:04.869+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:04.890 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd37497 2024-04-24T16:00:04.890 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:04.890 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:04.891 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:04.891 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:04.893 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:04.894 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:04.894 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd37497 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.577811237999999, 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.57665956, 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.561639776, 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.540707437999998, 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.540085483999999, 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.512100963000002, 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.498785272999999, 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.486073472000001, 2024-04-24T16:00:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:8 3.3 e44/15 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.448164302999999, 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0083961290000000004, 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448404+0000", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452217+0000", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038123480000000001 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452241+0000", 2024-04-24T16:00:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4261e-05 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452253+0000", 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2130000000000001e-05 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452299+0000", 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6060999999999998e-05 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452328+0000", 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.915e-05 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456722+0000", 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043940510000000004 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456803+0000", 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0901000000000004e-05 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.427484567, 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.408049111, 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.395569128999998, 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.268134080999999, 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.267447334, 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.26629286, 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.262645435, 2024-04-24T16:00:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.242685344000002, 2024-04-24T16:00:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.241277435000001, 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.240641386, 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.056785875999999, 2024-04-24T16:00:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:04.990 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:04.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:04.992 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:04.995+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:05.000 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:05.000 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:05.000 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:05.000 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:05.000 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:05.001 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:05.056 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38626 2024-04-24T16:00:05.056 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38626 2024-04-24T16:00:05.056 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38626 -C rbd-nbd 2024-04-24T16:00:05.061 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:05.061 INFO:tasks.workunit.client.0.smithi122.stdout: 38626 ? 00:00:00 rbd-nbd 2024-04-24T16:00:05.061 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T16:00:05.062 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:05.062 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:05.062 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:05.062 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:05.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:05.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:05.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T16:00:05.092 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:05.095+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:05.192 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:05.193 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:05.196+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:05.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:05.295 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:05.297+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:05.319 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:00:05.355 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:05.355 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:05.355 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:05.355 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:05.356 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:05.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:05.409 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:05.412+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:05.496 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:05.499+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:05.530 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-04-24T16:00:05.530 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-04-24T16:00:05.530 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.454321 s, 148 MB/s 2024-04-24T16:00:05.531 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false timeout 10 rbd bench testrbdnbd37497 --io-type write --io-size=1024 --io-total=1024 2024-04-24T16:00:05.531 INFO:tasks.workunit.client.0.smithi122.stderr:+ timeout 10 rbd bench testrbdnbd37497 --io-type write --io-size=1024 --io-total=1024 2024-04-24T16:00:05.562 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:05.562 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:05.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:05.562 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:05.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.257698915999999, 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.248078851999999, 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.227678285, 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:05.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.214746482999999, 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:05.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:05.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.214712731999999, 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:05.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:05.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.213190321999999, 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.585 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:05.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.207779206000001, 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.589 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:05.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.200030687999998, 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:05.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:05.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:05.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.197900664999999, 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.598 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:05.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:05.602 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.182197708, 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:05.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:05.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.182169721000001, 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:05.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.165398645, 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:05.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:05.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:05.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.160055120999999, 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.147844876000001, 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.627 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.140409711, 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:05.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:05.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:05.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:57 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.139463534000001, 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038713490000000003, 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57 2024-04-24T16:00:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424097+0000", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424102+0000", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3149999999999999e-06 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424118+0000", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5625999999999999e-05 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424125+0000", 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2960000000000001e-06 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424490+0000", 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000364781 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424569+0000", 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8584000000000005e-05 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424869+0000", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030080800000000001 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430343+0000", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054732189999999997 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462762+0000", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032419199000000003 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462789+0000", 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7463000000000001e-05 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462811+0000", 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2048999999999998e-05 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:59 3.2 3:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.638 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.092263338999999, 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039387966000000003, 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 59 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471297+0000", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479102+0000", 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078047949999999998 2024-04-24T16:00:05.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479124+0000", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2449000000000001e-05 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479132+0000", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6190000000000002e-06 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479193+0000", 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1668999999999995e-05 2024-04-24T16:00:05.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510682+0000", 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031488457999999997 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510686+0000", 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2150000000000003e-06 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.642 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.956153812, 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:05.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:05.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.951917885, 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:05.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:05.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.651 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.945078119000001, 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:05.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:05.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:05.656 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:05.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:05.658 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:05.660+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:05.659 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:05.572+0000 7f6a44d72700 -1 librbd::exclusive_lock::ImageDispatch: 0x7f6a20004b10 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-04-24T16:00:05.659 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to flush: (30) Read-only file system 2024-04-24T16:00:05.659 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:05.572+0000 7f6a44d72700 -1 librbd::io::AioCompletion: 0x55a2d4ae26f0 fail: (30) Read-only file system 2024-04-24T16:00:05.659 INFO:tasks.workunit.client.0.smithi122.stderr:bench failed: (30) Read-only file system 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 38626 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38626 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:05.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:05.758 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:05.761+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:05.846 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:05.847 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:05.847 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:05.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:05.859 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:05.862+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:05.961 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:05.964+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:05.997 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:05.997 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:05.997 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:05.997 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:05.997 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:06.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:06.092 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:06.095+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:06.153 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:06.153 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38626\b' 2024-04-24T16:00:06.153 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38626\b' 2024-04-24T16:00:06.163 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:06.165+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:06.189 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:06.190 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:06.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:06.190 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T16:00:06.190 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:06.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:06.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:06.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38626\b' 2024-04-24T16:00:06.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38626\b' 2024-04-24T16:00:06.200 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:06.200 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:06.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:06.264 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:06.266+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:06.334 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:06.334 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:06.334 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:06.334 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:06.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:06.365 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:06.368+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:06.465 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:06.468+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:06.508 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:06.508 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:06.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:06.566 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:06.569+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.643 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.333403303000001, 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:06.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.325437272999999, 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:06.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:06.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.325403078000001, 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:06.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:06.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.325384440000001, 2024-04-24T16:00:06.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:06.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:06.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:06.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.318061703000001, 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.666 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:06.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:06.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.671 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.29640268, 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:06.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:06.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.292154046, 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:06.681 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.292119528000001, 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:06.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:06.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.261958423999999, 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.210863485000001, 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.157344973000001, 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.034004812999999, 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.017135358000001, 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 18.008918933, 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.995378785, 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:06.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:06.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.993494116000001, 2024-04-24T16:00:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:06.720 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.972885857000001, 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.962964357000001, 2024-04-24T16:00:06.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:06.729 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.958316099000001, 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 17.956822308, 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:06.733 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:06.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:06.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.737 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:06.738 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:06.738 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd bench testrbdnbd37497 --io-type write --io-size=1024 --io-total=1024 2024-04-24T16:00:06.739 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:06.741+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:06.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:06.743 INFO:tasks.workunit.client.0.smithi122.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-04-24T16:00:06.745 INFO:tasks.workunit.client.0.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-04-24T16:00:06.745 INFO:tasks.workunit.client.0.smithi122.stdout:elapsed: 0 ops: 1 ops/sec: 500.012 bytes/sec: 500 KiB/s 2024-04-24T16:00:06.752 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:06.752 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:06.753 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:06.754 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:06.754 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:06.756 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:06.756 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:06.756 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:06.839 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:06.842+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:06.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:06.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:06.850 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:06.850 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:06.850 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:06.850 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.5095.0:55 3.3 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [sparse-read 3731456~86016] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T16:00:06.867659+0000", 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.001320395, 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013411149999999999, 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.866 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.5095", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1869509651", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867659+0000", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867659+0000", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867658+0000", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867660+0000", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.387e-06 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867663+0000", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8820000000000001e-06 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867672+0000", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8920000000000006e-06 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.868861+0000", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011898639999999999 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:06.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.868912+0000", 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0754999999999998e-05 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.5095.0:56 3.3 3.d55af4b7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T16:00:06.867674+0000", 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.001304983, 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013698860000000001, 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.5095", 2024-04-24T16:00:06.870 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1869509651", 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867674+0000", 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867674+0000", 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867674+0000", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867675+0000", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.407e-06 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867679+0000", 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9369999999999997e-06 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867681+0000", 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.04e-06 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.5095.0:57 3.3 3.d55af4b7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-24T16:00:06.873 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T16:00:06.867683+0000", 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0012957890000000001, 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013917510000000001, 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.5095", 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1869509651", 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 57 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867683+0000", 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867683+0000", 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867683+0000", 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867684+0000", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2470000000000001e-06 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867687+0000", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.92e-06 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867689+0000", 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9139999999999998e-06 2024-04-24T16:00:06.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.5095.0:58 3.3 3.d55af4b7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T16:00:06.867691+0000", 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.001287692, 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014129399999999999, 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.5095", 2024-04-24T16:00:06.877 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1869509651", 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867691+0000", 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867691+0000", 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867691+0000", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867698+0000", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8560000000000001e-06 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867701+0000", 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1760000000000002e-06 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:06.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T16:00:06.867703+0000", 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-06 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 4 2024-04-24T16:00:06.880 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:06.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:06.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:06.905 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38751 2024-04-24T16:00:06.905 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38751 2024-04-24T16:00:06.905 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38751 -C rbd-nbd 2024-04-24T16:00:06.910 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:06.910 INFO:tasks.workunit.client.0.smithi122.stdout: 38751 ? 00:00:00 rbd-nbd 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd37497 38751 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd37497 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38751 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:06.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:06.912 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:06.913 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:06.913 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:06.913 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:06.967 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:06.969+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:07.041 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:07.043+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:07.045 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:07.047 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:07.047 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:07.047 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:07.047 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:07.047 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:07.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:07.064 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:07.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:07.103 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:07.122 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:07.141 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:07.144+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:07.142 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:07.220 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.220 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:07.271 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:07.274+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:07.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:00:07.379 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:07.382+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:07.395 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.395 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:07.444 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:07.447+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:07.452 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:07.452 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:07.452 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T16:00:07.452 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-04-24T16:00:07.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-24T16:00:07.494 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:07.494 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38751\b' 2024-04-24T16:00:07.495 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38751\b' 2024-04-24T16:00:07.532 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.532 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.535 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:07.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:07.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38751\b' 2024-04-24T16:00:07.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38751\b' 2024-04-24T16:00:07.541 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:07.541 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:07.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:07.544 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:07.546+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:07.644 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:07.647+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:07.666 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.666 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:07.746 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:07.748+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:07.814 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.814 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:07.869 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:07.872+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:07.948 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:07.951+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:07.963 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:07.963 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:07.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:08.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:08.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@snap 2024-04-24T16:00:08.058 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:08.061+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:08.149 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:08.153+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:08.157 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:08.157 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:08.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:08.251 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:08.254+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:00:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-24T16:00:08.335 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T16:00:08.346 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497@snap 2024-04-24T16:00:08.346 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:08.346 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:08.346 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:08.347 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:08.349 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:08.349 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:08.349 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497@snap 2024-04-24T16:00:08.352 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:08.355+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:08.358 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:08.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:08.448 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:08.448 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:08.449 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:08.449 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:08.449 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:08.449 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:08.454 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:08.457+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:08.504 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38859 2024-04-24T16:00:08.504 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38859 2024-04-24T16:00:08.504 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38859 -C rbd-nbd 2024-04-24T16:00:08.509 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:08.509 INFO:tasks.workunit.client.0.smithi122.stdout: 38859 ? 00:00:00 rbd-nbd 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd37497@snap 38859 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd37497@snap 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38859 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd37497@snap 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:08.510 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:08.511 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:08.511 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:08.512 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:08.512 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:08.512 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd37497@snap 2024-04-24T16:00:08.553 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:08.553 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:08.553 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:08.556+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:08.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:08.654 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:08.657+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:08.746 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:08.746 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:08.755 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:08.758+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:08.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:08.856 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:08.859+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:08.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:08.957 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:08.960+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:09.044 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:09.044 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:09.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:09.058 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:09.061+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:09.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:09.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38859\b' 2024-04-24T16:00:09.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38859\b' 2024-04-24T16:00:09.134 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:09.134 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:09.135 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38859\b' 2024-04-24T16:00:09.135 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38859\b' 2024-04-24T16:00:09.140 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:09.140 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:09.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:09.159 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:09.162+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:09.195 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:09.195 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:09.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:09.260 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:09.263+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:09.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:09.331 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:09.345 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:09.345 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:09.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:09.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:09.371 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:09.391 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:09.396 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:09.399+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:09.410 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:09.411 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:09.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:09.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:09.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:09.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:09.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:09.462 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:09.465+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:09.519 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:09.519 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:09.519 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:09.519 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:09.519 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:09.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:09.569 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:09.572+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:09.641 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:09.641 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/testrbdnbd37497 2024-04-24T16:00:09.641 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T16:00:09.664 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:09.666+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:09.682 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=4 2024-04-24T16:00:09.682 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd37497 2024-04-24T16:00:09.682 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:09.682 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:09.683 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:09.683 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:09.685 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:09.685 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:09.685 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd37497 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.694 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.384451434999999, 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:09.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.700 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.376485405, 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:09.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:09.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:09.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.376451209999999, 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.376432571999999, 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:09.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.369109835, 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:09.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:09.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:09.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:09.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.347450812000002, 2024-04-24T16:00:09.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.343202177999999, 2024-04-24T16:00:09.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:09.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:09.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:09.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.343167659999999, 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:09.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.313006556000001, 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.261911616999999, 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.208393104999999, 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.085052945000001, 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:09.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.749 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.068183489999999, 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:09.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:09.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:09.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.059967064999999, 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:09.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:09.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.046426917000002, 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.763 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.044542247999999, 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:09.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.023933989, 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:09.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:09.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.014012488999999, 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.009364230999999, 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.781 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.007870440000001, 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:09.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:09.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:09.786 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:09.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:09.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:09.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:09.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:09.789 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:09.789 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:09.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:09.789 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:09.790+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:09.843 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=38969 2024-04-24T16:00:09.843 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 38969 2024-04-24T16:00:09.843 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 38969 -C rbd-nbd 2024-04-24T16:00:09.848 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:09.848 INFO:tasks.workunit.client.0.smithi122.stdout: 38969 ? 00:00:00 rbd-nbd 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--snap-id 4 testrbdnbd37497' 38969 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--snap-id 4 testrbdnbd37497' 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=38969 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd37497 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:09.849 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:09.850 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:09.850 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:09.851 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:09.851 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:09.851 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd37497 2024-04-24T16:00:09.889 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:09.891+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:09.954 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:09.954 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:09.954 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:09.955 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:09.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:10.006 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:10.008+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:10.091 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:10.093+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:10.128 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:10.128 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:10.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:10.191 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:10.194+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.958759101999998, 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.264 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:10.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.957612315999999, 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:10.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:10.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:10.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.941760194, 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.275 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:10.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.918995562999999, 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.281 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.908877207, 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:10.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:10.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:10.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.880934092, 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.291 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.880916266, 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:10.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.871298868, 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.867569957000001, 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.305 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.841685894000001, 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:10.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.785464254000001, 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:10.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:10.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.319 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.785443143999998, 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:10.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:10.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.672568299000002, 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.325 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:10.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.663150528999999, 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.329 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:10.333 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.646634363, 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:10.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:10.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.337 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.646077946999998, 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:10.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.596847542999999, 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:10.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:10.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:10.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.346 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.585169715999999, 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:10.350 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.583821957000001, 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:10.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:10.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:10.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 21.575694377000001, 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.355 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:10.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:10.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:10.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.359 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:10.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T16:00:10.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:10.362 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:10.363+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:10.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:10.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^38969\b' 2024-04-24T16:00:10.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^38969\b' 2024-04-24T16:00:10.462 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:10.464+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:10.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:10.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:10.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *38969\b' 2024-04-24T16:00:10.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *38969\b' 2024-04-24T16:00:10.488 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:10.488 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:10.526 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:10.526 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:10.526 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:10.526 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:10.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:10.576 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:10.579+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:10.664 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:10.667+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:10.698 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:10.699 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:10.699 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:10.699 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:10.699 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:10.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:10.765 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:10.768+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:10.833 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:10.833 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.510962391, 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:10.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:10.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.840 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.509157811000001, 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:10.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.509126443, 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:10.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:10.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.485402716999999, 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:10.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:10.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.457544979000001, 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:10.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:10.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:10.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:10.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.452915474000001, 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.860 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:10.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:10.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.446107931, 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:10.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.415567704000001, 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:10.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:10.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:10.874 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.399928024000001, 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:10.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.360752775000002, 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.879 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:10.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:10.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.348134654999999, 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.225738518, 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:10.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:10.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:10.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.212209309999999, 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.201474982000001, 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.898 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.187178839000001, 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.904 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:10.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:10.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:10.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.154951746999998, 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:10.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:10.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.148855906000001, 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:10.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.920 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:10.921 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:10.921 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.145401349, 2024-04-24T16:00:10.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:10.921 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:10.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 22.109957227999999, 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 15.058497463, 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:10.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:10.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.933 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:10.934 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:10.934 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:10.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:10.936 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:10.938+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:10.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:10.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.036 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:11.039+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:11.085 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:11.085 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:11.085 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:11.085 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:11.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:11.141 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:11.144+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:11.235 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:11.235 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:11.235 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:11.235 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:11.235 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:11.238 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:11.241+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:11.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T16:00:11.340 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:11.342+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:11.347 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T16:00:11.355 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.355 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:11.355 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:11.357 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:11.357 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:11.359 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:11.359 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:11.359 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:11.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.118298119999999, 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:11.440 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.117146441999999, 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:11.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:11.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.444 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.102126658, 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:11.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.448 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.081194320000002, 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:11.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:11.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:11.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.452 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.080572365999998, 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:11.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:11.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.456 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.052587845000001, 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:11.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:11.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.039272154999999, 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:11.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:11.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.464 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.026560354000001, 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:11.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:11.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:11.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:8 3.3 e44/15 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.988651184999998, 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0083961290000000004, 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448407+0000", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.448404+0000", 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452217+0000", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038123480000000001 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452241+0000", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4261e-05 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452253+0000", 2024-04-24T16:00:11.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2130000000000001e-05 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452299+0000", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6060999999999998e-05 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.452328+0000", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.915e-05 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456722+0000", 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043940510000000004 2024-04-24T16:00:11.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.456803+0000", 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0901000000000004e-05 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.967971449, 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.475 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:11.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:11.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.948535993, 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.479 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:11.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:11.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.936056011000002, 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.808620962999999, 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:11.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.807934216, 2024-04-24T16:00:11.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:11.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:11.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.497 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.806779742, 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:11.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:11.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:11.501 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.803132316999999, 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.783172226000001, 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.781764317, 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.781128268, 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.597272757999999, 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:11.524 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:11.526 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:11.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:11.527 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:11.528+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:11.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39077 2024-04-24T16:00:11.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39077 2024-04-24T16:00:11.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39077 -C rbd-nbd 2024-04-24T16:00:11.532 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:11.532 INFO:tasks.workunit.client.0.smithi122.stdout: 39077 ? 00:00:00 rbd-nbd 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/ns/testrbdnbd37497@snap 39077 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39077 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:11.533 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:11.534 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:11.534 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:11.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:11.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:11.536 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd37497@snap 2024-04-24T16:00:11.583 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:11.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:11.624 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:11.626 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:11.629+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:11.644 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:11.663 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:11.683 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:11.684 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:11.684 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:11.684 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:11.684 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:11.684 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:11.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:11.707 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:11.707 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:11.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:11.727 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:11.730+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:11.828 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:11.831+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:11.901 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:11.902 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:11.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:11.929 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:11.932+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:12.030 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:12.033+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:12.096 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.096 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:12.114 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:12.114 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39077\b' 2024-04-24T16:00:12.114 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39077\b' 2024-04-24T16:00:12.148 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:12.151+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:12.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:12.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:12.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39077\b' 2024-04-24T16:00:12.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39077\b' 2024-04-24T16:00:12.166 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:12.166 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:12.232 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:12.235+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:12.289 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.289 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:12.342 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:12.345+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:12.435 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:12.438+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:12.482 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.483 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:12.538 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:12.541+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:12.632 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.632 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.636 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:12.640+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:12.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:12.667 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:12.668 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/ns/testrbdnbd37497 2024-04-24T16:00:12.668 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T16:00:12.711 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=5 2024-04-24T16:00:12.712 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd37497 2024-04-24T16:00:12.712 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:12.712 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:12.713 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:12.713 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:12.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:12.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:12.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd37497 2024-04-24T16:00:12.738 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:12.740+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:12.779 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.779 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:12.826 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:12.827 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-04-24T16:00:12.827 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:12.827 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-04-24T16:00:12.827 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:12.827 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:12.839 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:12.842+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:12.886 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39172 2024-04-24T16:00:12.886 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39172 2024-04-24T16:00:12.886 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39172 -C rbd-nbd 2024-04-24T16:00:12.890 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:12.890 INFO:tasks.workunit.client.0.smithi122.stdout: 39172 ? 00:00:00 rbd-nbd 2024-04-24T16:00:12.891 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd37497' 39172 2024-04-24T16:00:12.891 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd37497' 2024-04-24T16:00:12.891 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39172 2024-04-24T16:00:12.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd37497 2024-04-24T16:00:12.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:12.892 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:12.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:12.892 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:12.894 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:12.894 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:12.894 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd37497 2024-04-24T16:00:12.940 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:12.943+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:12.952 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:12.952 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:12.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:13.041 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:13.044+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:13.085 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.085 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:13.142 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:13.145+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:13.218 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.219 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:13.272 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:13.275+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:13.343 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:13.346+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:13.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:13.346 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:13.346 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-24T16:00:13.346 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:00:13.346 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:00:13.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-24T16:00:13.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:00:13.352 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.353 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:13.444 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:13.447+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:13.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:13.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39172\b' 2024-04-24T16:00:13.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39172\b' 2024-04-24T16:00:13.494 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:13.495 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T16:00:13.495 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:13.515 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:13.515 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:13.515 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39172\b' 2024-04-24T16:00:13.516 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39172\b' 2024-04-24T16:00:13.520 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:13.521 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:13.525 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.525 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:13.551 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:13.554+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:13.646 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:13.648+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:13.658 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.658 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:13.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:13.756 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:13.758+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:13.832 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:13.832 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:13.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:13.847 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:13.850+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:13.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:13.948 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:13.951+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:14.006 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:14.006 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:14.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:14.022 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:14.022 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd37497 2024-04-24T16:00:14.022 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:14.022 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:14.023 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:14.024 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:14.026 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:14.026 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:14.026 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd37497 2024-04-24T16:00:14.058 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:14.060+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:14.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:14.113 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:14.126 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:14.126 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-04-24T16:00:14.127 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:14.127 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-04-24T16:00:14.127 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:14.127 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:14.134 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:14.150 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:14.153+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:14.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:14.164 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:14.164 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:14.164 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:14.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:14.180 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39245 2024-04-24T16:00:14.180 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39245 2024-04-24T16:00:14.181 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39245 -C rbd-nbd 2024-04-24T16:00:14.186 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:14.186 INFO:tasks.workunit.client.0.smithi122.stdout: 39245 ? 00:00:00 rbd-nbd 2024-04-24T16:00:14.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd37497' 39245 2024-04-24T16:00:14.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd37497' 2024-04-24T16:00:14.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39245 2024-04-24T16:00:14.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd37497 2024-04-24T16:00:14.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:14.188 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:14.188 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:14.188 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:14.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:14.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:14.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd37497 2024-04-24T16:00:14.260 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:14.263+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:00:14.351 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:14.354+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:14.361 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:14.361 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:14.361 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:14.361 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:14.361 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:14.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:14.452 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:14.455+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:14.553 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:14.556+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:14.556 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:14.556 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:14.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.238612474, 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:14.561 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.237460796000001, 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.562 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.222441012000001, 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.201508673999999, 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.20088672, 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.172902198999999, 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.159586509, 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:14.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.146874707999999, 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:14.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.088285803000002, 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:14.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.068850347000001, 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:14.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.056370364999999, 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:14.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:14.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.928935317000001, 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:14.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:14.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:14.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.928248570000001, 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.613 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:14.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:14.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:14.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.927094096000001, 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:14.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.923446671000001, 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:14.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.903486579999999, 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:14.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:14.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:14.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.902078671000002, 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:14.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:14.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.901442622000001, 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.643 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.717587112, 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:14.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:14.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:14.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.647 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.0788435249999999, 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:14.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:14.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:14.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:14.652 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:14.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:14.694 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:14.697+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:14.755 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:14.758+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:14.769 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:14.769 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39245\b' 2024-04-24T16:00:14.769 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39245\b' 2024-04-24T16:00:14.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:14.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:14.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39245\b' 2024-04-24T16:00:14.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39245\b' 2024-04-24T16:00:14.815 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:14.815 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:14.838 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:14.838 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:14.838 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:14.838 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:14.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:14.894 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:14.897+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:14.958 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:14.961+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:14.988 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:14.988 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:14.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:15.083 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:15.086+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:15.160 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:15.163+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:15.183 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.878908828, 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.184 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.869288764, 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:15.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.848888197000001, 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:15.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.835956395, 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:15.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:15.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.835922644, 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:15.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.834400234, 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.211 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:15.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:15.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.828989117999999, 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:15.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:15.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.821240599999999, 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:15.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.819110577, 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:15.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.803407620000002, 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:15.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.803379632999999, 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:15.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.243 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.786608557000001, 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:15.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:15.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:15.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.781265033, 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.250 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:15.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:15.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:15.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.769054787999998, 2024-04-24T16:00:15.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:15.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.761619623000001, 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.263 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:15.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:57 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.760673445999998, 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038713490000000003, 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:15.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424097+0000", 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424102+0000", 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3149999999999999e-06 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424118+0000", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5625999999999999e-05 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424125+0000", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2960000000000001e-06 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424490+0000", 2024-04-24T16:00:15.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000364781 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424569+0000", 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8584000000000005e-05 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424869+0000", 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030080800000000001 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430343+0000", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054732189999999997 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462762+0000", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032419199000000003 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462789+0000", 2024-04-24T16:00:15.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7463000000000001e-05 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462811+0000", 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2048999999999998e-05 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:59 3.2 3:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.275 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.713473251, 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039387966000000003, 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 59 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471297+0000", 2024-04-24T16:00:15.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479102+0000", 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078047949999999998 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479124+0000", 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2449000000000001e-05 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479132+0000", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6190000000000002e-06 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479193+0000", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1668999999999995e-05 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510682+0000", 2024-04-24T16:00:15.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031488457999999997 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510686+0000", 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2150000000000003e-06 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.280 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.577363724000001, 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:15.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:15.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.285 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.573127797000001, 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:15.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:15.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.566288030999999, 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:15.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:15.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:15.296 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:15.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:15.299 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:15.301+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:15.317 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd37497 --snap snap 2024-04-24T16:00:15.317 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:15.317 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:15.318 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:15.318 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:15.320 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:15.320 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:15.320 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd37497 --snap snap 2024-04-24T16:00:15.399 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:15.401+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:15.485 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:15.485 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:15.485 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:15.485 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:15.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:15.499 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:15.502+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:15.600 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:15.603+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:15.627 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:15.636 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:15.636 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:15.636 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:15.636 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:15.636 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:15.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:15.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39318 2024-04-24T16:00:15.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39318 2024-04-24T16:00:15.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39318 -C rbd-nbd 2024-04-24T16:00:15.688 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:15.688 INFO:tasks.workunit.client.0.smithi122.stdout: 39318 ? 00:00:00 rbd-nbd 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd37497 --snap snap' 39318 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd37497 --snap snap' 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39318 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd37497 --snap snap 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:15.689 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:15.690 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:15.690 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:15.692 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:15.692 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:15.692 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd37497 --snap snap 2024-04-24T16:00:15.701 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:15.704+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:15.801 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:15.804+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:15.831 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:15.831 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:15.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:15.832 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T16:00:15.832 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:15.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:15.902 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:15.905+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:15.979 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:15.980 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:15.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:16.003 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:16.006+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:16.104 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:16.106+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:16.112 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:16.113 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:16.113 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:16.113 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:16.113 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:16.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:16.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:16.213 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:16.216+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:16.275 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:16.275 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39318\b' 2024-04-24T16:00:16.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39318\b' 2024-04-24T16:00:16.287 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:16.287 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:16.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:16.287 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.977664553, 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:16.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:16.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:16.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.294 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.969698523000002, 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:16.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.299 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.969664328, 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:16.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.96964569, 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:16.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.962322953000001, 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:16.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.940663929999999, 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:16.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.936415296, 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:16.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:16.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.936380778, 2024-04-24T16:00:16.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:16.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.906219673999999, 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:16.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:16.337 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.855124735, 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:16.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.342 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.801606223, 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:16.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:16.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.678266062999999, 2024-04-24T16:00:16.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:16.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.352 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.661396608, 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:16.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:16.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:16.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:16.359 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.653180183, 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:16.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:16.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:16.364 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.639640034999999, 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:16.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:16.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:16.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.637755366, 2024-04-24T16:00:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.617147107000001, 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:16.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:16.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.381 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.607225607, 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:16.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:16.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.386 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.602577349000001, 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.601083557999999, 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:16.398 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:16.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:00:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39318\b' 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39318\b' 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:16.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:16.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:16.403 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:16.402+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:16.414 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:16.435 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:16.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:16.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:16.500 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:16.503+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:16.566 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:16.566 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:16.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:16.620 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:16.623+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:16.700 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:16.700 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:16.703 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:16.705+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:16.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:16.803 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:16.806+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:16.825 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:16.825 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:16.825 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:16.825 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:16.826 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:16.827 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:16.829 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:16.829 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:16.829 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:16.873 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:16.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:16.924 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:16.927+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:16.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:16.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:16.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:16.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:16.928 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:16.928 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:16.984 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39391 2024-04-24T16:00:16.984 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39391 2024-04-24T16:00:16.984 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39391 -C rbd-nbd 2024-04-24T16:00:16.989 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:16.989 INFO:tasks.workunit.client.0.smithi122.stdout: 39391 ? 00:00:00 rbd-nbd 2024-04-24T16:00:16.989 INFO:tasks.workunit.client.0.smithi122.stderr:+ pid=39391 2024-04-24T16:00:16.990 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd37497 2024-04-24T16:00:16.990 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:16.990 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:16.991 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:16.991 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:16.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:16.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:16.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd37497 2024-04-24T16:00:17.005 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:17.008+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:17.047 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.047 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:17.098 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-04-24T16:00:17.099 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-04-24T16:00:17.099 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:17.099 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-04-24T16:00:17.099 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:17.099 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T16:00:17.142 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:17.145+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:17.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39445 2024-04-24T16:00:17.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39445 2024-04-24T16:00:17.162 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39445 -C rbd-nbd 2024-04-24T16:00:17.166 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:17.166 INFO:tasks.workunit.client.0.smithi122.stdout: 39391 ? 00:00:00 rbd-nbd 2024-04-24T16:00:17.167 INFO:tasks.workunit.client.0.smithi122.stdout: 39445 ? 00:00:00 rbd-nbd 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/ns/testrbdnbd37497 39445 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/ns/testrbdnbd37497 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39445 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd37497 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:17.168 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:17.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:17.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:17.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd37497 2024-04-24T16:00:17.211 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:17.213+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:17.222 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.222 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:17.316 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:17.319+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:00:17.397 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.397 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:17.451 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:17.454+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:17.513 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:17.516+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:17.554 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.554 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:17.650 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:17.653+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:17.716 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:17.718+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:17.747 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.747 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.753 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:17.753 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39445\b' 2024-04-24T16:00:17.753 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39445\b' 2024-04-24T16:00:17.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:17.804 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:17.805 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:17.805 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39445\b' 2024-04-24T16:00:17.805 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39445\b' 2024-04-24T16:00:17.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:17.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:17.842 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:17.845+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:17.919 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:17.922+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:17.940 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:17.940 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:17.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:18.036 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:18.039+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:18.121 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:18.124+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:18.134 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:18.134 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:18.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:18.222 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:18.225+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/testrbdnbd37497 39391 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/testrbdnbd37497 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39391 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd37497 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:18.312 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:18.313 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:18.313 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:18.315 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:18.315 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:18.315 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd37497 2024-04-24T16:00:18.323 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:18.326+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:18.327 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:18.327 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:18.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:18.424 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:18.427+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:18.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:18.510 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:18.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T16:00:18.523 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:18.523 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:18.525 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:18.528+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:18.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:18.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:18.626 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:18.628+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:18.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:18.656 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:18.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:18.699 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:18.719 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:18.722 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:18.722 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:18.726 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:18.729+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:18.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:18.827 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:18.830+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:18.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:18.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39391\b' 2024-04-24T16:00:18.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39391\b' 2024-04-24T16:00:18.914 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:18.914 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:18.914 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:18.914 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:18.914 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:18.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:18.928 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:18.931+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:18.943 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:18.944 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:18.944 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39391\b' 2024-04-24T16:00:18.944 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39391\b' 2024-04-24T16:00:18.952 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:18.952 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:19.029 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:19.031+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:19.063 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-04-24T16:00:19.064 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:19.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:19.130 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:19.132+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:19.211 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:19.211 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:19.211 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:19.211 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:19.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:19.231 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:19.234+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:19.331 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:19.334+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:19.354 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T16:00:19.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:00:19.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T16:00:19.386 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:19.387 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:19.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:19.452 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:19.455+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:19.453 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' rbd = rbd ']' 2024-04-24T16:00:19.453 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd37497 2024-04-24T16:00:19.453 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:19.454 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:19.455 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:19.455 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:19.457 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:19.457 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:19.458 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd37497 2024-04-24T16:00:19.533 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:19.535+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:19.566 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.255715227, 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.247749197000001, 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.247715002, 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:19.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.247696363999999, 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.240373627, 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.218714603999999, 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:19.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:19.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.214465969999999, 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.599 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:19.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:19.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:19.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.214431451999999, 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:19.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.608 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.184270347999998, 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.133175409, 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.079656897, 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:19.619 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.620 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.620 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:19.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:19.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:19.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:19.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:19.628 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.956316737000002, 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:19.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:19.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:19.637 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.939447282, 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:19.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:19.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:19.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:19.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.649 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.931230856999999, 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.650 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:19.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:19.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.917690708999999, 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.657 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:19.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:19.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:19.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.662 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.91580604, 2024-04-24T16:00:19.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:19.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.672 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.895197781, 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:19.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:19.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:19.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:19.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.885276280999999, 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.683 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:19.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.880628023, 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:19.702 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.879134231999998, 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:19.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:19.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:19.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:19.708 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:19.710 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:19.713+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:19.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:19.712 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:19.712 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39559 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39559 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39559 -C rbd-nbd 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:19.713 INFO:tasks.workunit.client.0.smithi122.stdout: 39559 ? 00:00:00 rbd-nbd 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd37497 39559 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd37497 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39559 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:19.715 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:19.716 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:19.810 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:19.813+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:19.876 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:19.876 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:19.876 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:19.876 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:19.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:19.930 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:19.933+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:20.009 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:20.010 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:20.012 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:20.015+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:20.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:20.113 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:20.116+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.185 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.880335809000002, 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:20.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:20.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.879189022999999, 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:20.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:20.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.863336901, 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:20.195 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:20.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:20.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.840572269999999, 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.199 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.830453914, 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:20.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:20.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.802510799, 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.802492973, 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:20.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:20.217 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.792875575, 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.221 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.789146664, 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.763262601000001, 2024-04-24T16:00:20.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.707040961000001, 2024-04-24T16:00:20.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.231 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:20.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:20.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:20.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.707019850999998, 2024-04-24T16:00:20.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.594145006000002, 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:20.241 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:20.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.584727235999999, 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:20.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.249 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.56821107, 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:20.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:20.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.567654654000002, 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:20.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:20.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.518424249999999, 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:20.261 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.506746422999999, 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:20.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:20.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.505398664000001, 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:20.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 31.497271084000001, 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.280 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:20.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:20.283 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:20.284+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:20.283 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:20.283 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39559\b' 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39559\b' 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39559\b' 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39559\b' 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:20.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T16:00:20.359 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:00:20.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T16:00:20.414 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:20.416+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:20.448 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:20.448 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:20.448 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:20.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:20.501 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:20.504+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:20.581 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:20.581 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:20.581 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:20.583 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:20.585+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:20.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:20.684 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:20.687+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:20.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:20.756 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.434218301000001, 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.432413721000003, 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.432382353000001, 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:20.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:20.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.408658627000001, 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:20.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:20.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:20.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.380800889, 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:20.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:20.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:20.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.376171384000003, 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:20.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.782 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.369363841000002, 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:20.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:20.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.338823613999999, 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:20.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:20.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:20.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.323183933999999, 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:20.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:20.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.284008685000003, 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.271390564999997, 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.148994428000002, 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.13546522, 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.124730892000002, 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.110434748999999, 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:20.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.078207657, 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:20.822 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:20.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:20.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:20.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.072111816000003, 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:20.827 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:20.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:20.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.068657258999998, 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:20.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.033213138000001, 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:20.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:20.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:20.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 24.981753373, 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:20.840 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:20.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:20.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:20.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:20.844 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:20.846 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:20.848+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:20.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global set global rbd_default_pool new_default_pool37497 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd create --size 10M testrbdnbd37497 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd37497 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:20.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd37497 2024-04-24T16:00:20.895 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:20.916 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:20.940 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:20.940 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid new_default_pool37497 2024-04-24T16:00:20.940 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=new_default_pool37497 2024-04-24T16:00:20.941 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:20.941 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:20.941 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool37497'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:20.946 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:20.949+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:20.959 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:20.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:20.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39696 2024-04-24T16:00:20.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39696 2024-04-24T16:00:20.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39696 -C rbd-nbd 2024-04-24T16:00:20.993 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:20.994 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:20.998 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:20.998 INFO:tasks.workunit.client.0.smithi122.stdout: 39696 ? 00:00:00 rbd-nbd 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd37497 39696 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd37497 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39696 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:20.999 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:21.000 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:21.000 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:21.001 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:21.001 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:21.001 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd37497 2024-04-24T16:00:21.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:21.054 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:21.056+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:21.145 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.146 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:21.149 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:21.152+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:21.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:21.250 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:21.252+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:21.338 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.338 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:21.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T16:00:21.360 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:21.362+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:21.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T16:00:21.459 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:21.461+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:21.501 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.501 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:21.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:21.553 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:21.556+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:21.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:21.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39696\b' 2024-04-24T16:00:21.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39696\b' 2024-04-24T16:00:21.626 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:21.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:21.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39696\b' 2024-04-24T16:00:21.627 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39696\b' 2024-04-24T16:00:21.632 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:21.632 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:21.654 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:21.657+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:21.691 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.692 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:21.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:21.756 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:21.758+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:21.843 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.843 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:21.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:21.857 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:21.859+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:21.957 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:21.960+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:21.993 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:21.993 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:22.058 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:22.061+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:22.133 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:22.133 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global rm global rbd_default_pool 2024-04-24T16:00:22.142 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.143 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:22.159 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:22.162+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:22.167 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:22.167 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:22.167 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:22.168 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:22.168 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:22.170 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:22.170 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:22.170 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:22.260 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:22.263+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:22.295 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.295 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:22.362 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:22.363+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:22.362 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T16:00:22.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T16:00:22.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T16:00:22.449 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.449 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:22.462 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:22.465+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:22.563 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:22.566+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:22.587 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.587 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:22.681 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:22.684+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:22.761 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.761 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.764 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:22.766+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:22.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:22.865 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:22.867+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:22.936 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:22.936 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:22.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:22.966 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:22.968+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:23.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:23.066 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:23.069+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:23.068 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:23.069 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:23.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:23.168 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:23.171+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:23.177 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:23.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:23.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:23.235 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:23.242 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:23.242 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:23.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:23.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:23.274 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:23.293 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:23.296+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:23.363 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T16:00:23.363 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T16:00:23.370 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:23.372+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:23.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T16:00:23.401 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:23.401 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:23.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:23.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:23.402 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:23.402 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:23.419 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:23.419 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:23.419 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:23.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:23.460 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39801 2024-04-24T16:00:23.461 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39801 2024-04-24T16:00:23.461 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39801 -C rbd-nbd 2024-04-24T16:00:23.466 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:23.466 INFO:tasks.workunit.client.0.smithi122.stdout: 39801 ? 00:00:00 rbd-nbd 2024-04-24T16:00:23.466 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo kill 39801 2024-04-24T16:00:23.467 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:23.467 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:23.467 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:23.468 INFO:tasks.workunit.client.0.smithi122.stderr:++ which kill 2024-04-24T16:00:23.469 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/kill 2024-04-24T16:00:23.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:23.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/kill 39801 2024-04-24T16:00:23.471 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:23.473+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:23.573 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:23.575+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:23.578 INFO:tasks.workunit.client.0.smithi122.stderr:++ seq 10 2024-04-24T16:00:23.591 INFO:tasks.workunit.client.0.smithi122.stderr:+ for i in `seq 10` 2024-04-24T16:00:23.591 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:23.591 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39801 *rbd *testrbdnbd37497' 2024-04-24T16:00:23.591 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39801 *rbd *testrbdnbd37497' 2024-04-24T16:00:23.594 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:23.594 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:23.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:23.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:23.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-04-24T16:00:23.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:23.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39801 *rbd *testrbdnbd37497' 2024-04-24T16:00:23.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39801 *rbd *testrbdnbd37497' 2024-04-24T16:00:23.674 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:23.674 INFO:tasks.workunit.client.0.smithi122.stderr:+ QUIESCE_HOOK=/tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:23.674 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.P3d5b6KQ4E/quiesce.sh rbd/testrbdnbd37497 2024-04-24T16:00:23.675 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:23.675 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:23.676 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:23.676 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:23.678 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:23.678 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:23.678 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.P3d5b6KQ4E/quiesce.sh rbd/testrbdnbd37497 2024-04-24T16:00:23.688 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:23.691+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:23.770 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.447927571000001, 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:23.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:23.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.446122991000003, 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:23.779 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:23.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:23.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.446091623000001, 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:23.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.422367897000001, 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:23.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.394510158999999, 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.389880654000002, 2024-04-24T16:00:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.383073111000002, 2024-04-24T16:00:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:23.804 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.352532883999999, 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:23.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.336893203999999, 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:23.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:23.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:23.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.297717955000003, 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.285099834999997, 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.817 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:23.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:23.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:23.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.162703698000001, 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:23.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:23.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.14917449, 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:23.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.138440162000002, 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:23.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.834 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.124144018999999, 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.091916927, 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:23.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:23.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:23.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.085821086000003, 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.845 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:23.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:23.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.082366528999998, 2024-04-24T16:00:23.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:23.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 35.046922408, 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:23.854 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:23.858 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.995462643, 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:23.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:23.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:23.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:23.862 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:23.864 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:23.865+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:23.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=39874 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 39874 2024-04-24T16:00:23.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 39874 -C rbd-nbd 2024-04-24T16:00:23.867 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:23.867 INFO:tasks.workunit.client.0.smithi122.stdout: 39874 ? 00:00:00 rbd-nbd 2024-04-24T16:00:23.867 INFO:tasks.workunit.client.0.smithi122.stderr:+ test '!' -e /tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:23.867 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd snap create rbd/testrbdnbd37497@quiesce1 2024-04-24T16:00:23.867 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@quiesce1 2024-04-24T16:00:23.880 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete...2024-04-24T16:00:23.883+0000 7f397affd700 -1 librbd::ImageWatcher: 0x7f395c007a60 failed to notify quiesce: (1) Operation not permitted 2024-04-24T16:00:23.880 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:23.883+0000 7f397affd700 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-04-24T16:00:23.881 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete...failed. 2024-04-24T16:00:23.881 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-04-24T16:00:23.889 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:23.889 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:23.889 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:23.889 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:23.890 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:23.890 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:23.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:23.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:23.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:23.924 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:23.925 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:23.925 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227033 s, 46.2 MB/s 2024-04-24T16:00:23.925 INFO:tasks.workunit.client.0.smithi122.stderr:+ touch /tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:23.926 INFO:tasks.workunit.client.0.smithi122.stderr:+ chmod +x /tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:23.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-04-24T16:00:23.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@quiesce1 2024-04-24T16:00:23.964 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:23.966+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:24.015 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:24.015 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:24.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:24.065 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:24.068+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:24.166 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:24.169+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:24.208 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:24.208 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:24.208 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:24.209 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:24.209 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:24.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:24.268 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:24.270+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.044486784999997, 2024-04-24T16:00:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.043335106999997, 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.028315323000001, 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.007382985, 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.006761031000003, 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.978776510000003, 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.965460819999997, 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.952749019000002, 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.894160114000002, 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.874724657999998, 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.862244676000003, 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.734809628000001, 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.734122880999998, 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.732968407000001, 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.729320981999997, 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.709360891000003, 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.707952982000002, 2024-04-24T16:00:24.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:24.437 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.707316933000001, 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.442 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.523461423000001, 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:24.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:24.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.884717836, 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.451 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:24.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:24.454 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T16:00:24.454 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T16:00:24.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:24.455 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:24.455+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:24.553 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:24.556+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:24.645 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:24.645 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:24.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.645 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3.7a3bf4ec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.27566463899999999, 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27569175000000001, 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for peered", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:00:24.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:00:24.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3.7a3bf4ec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.273439975, 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27357494900000001, 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for peered", 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:24.650 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:00:24.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:00:24.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 2 2024-04-24T16:00:24.654 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:24.655 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:24.657+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:24.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:24.756 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:24.758+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:24.839 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:24.839 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:24.839 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:24.839 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:24.839 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:24.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:24.861 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:24.861 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:24.862+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:24.862 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:24.875 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-04-24T16:00:24.372478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218694615,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":7,"num_read":3333,"num_read_kb":163652,"num_write":723,"num_write_kb":264751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":2223976,"kb_used_data":431620,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564007064,"statfs":{"total":579820584960,"available":577543233536,"internally_reserved":0,"allocated":441978880,"data_stored":439188531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":296,"num_read_kb":3255,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016107"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":86,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358229+0000","last_change":"2024-04-24T16:00:21.358408+0000","last_active":"2024-04-24T16:00:21.358210+0000","last_peered":"2024-04-24T16:00:21.053511+0000","last_clean":"2024-04-24T16:00:21.053511+0000","last_became_active":"2024-04-24T15:59:28.190640+0000","last_became_peered":"2024-04-24T15:59:28.190640+0000","last_unstale":"2024-04-24T16:00:21.358229+0000","last_undegraded":"2024-04-24T16:00:21.358229+0000","last_fullsized":"2024-04-24T16:00:21.358229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:50:16.740896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357547+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:20.357547+0000","last_peered":"2024-04-24T16:00:20.357547+0000","last_clean":"2024-04-24T16:00:20.357547+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:20.357547+0000","last_undegraded":"2024-04-24T16:00:20.357547+0000","last_fullsized":"2024-04-24T16:00:20.357547+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":90,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357208+0000","last_change":"2024-04-24T15:59:48.255715+0000","last_active":"2024-04-24T16:00:20.357208+0000","last_peered":"2024-04-24T16:00:20.357208+0000","last_clean":"2024-04-24T16:00:20.357208+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T16:00:20.357208+0000","last_undegraded":"2024-04-24T16:00:20.357208+0000","last_fullsized":"2024-04-24T16:00:20.357208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053524+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:21.053524+0000","last_peered":"2024-04-24T16:00:21.053524+0000","last_clean":"2024-04-24T16:00:21.053524+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:21.053524+0000","last_undegraded":"2024-04-24T16:00:21.053524+0000","last_fullsized":"2024-04-24T16:00:21.053524+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":43,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.082668+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:21.082668+0000","last_peered":"2024-04-24T16:00:21.082668+0000","last_clean":"2024-04-24T16:00:21.082668+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:21.082668+0000","last_undegraded":"2024-04-24T16:00:21.082668+0000","last_fullsized":"2024-04-24T16:00:21.082668+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":86,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357003+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:20.357003+0000","last_peered":"2024-04-24T16:00:20.357003+0000","last_clean":"2024-04-24T16:00:20.357003+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:20.357003+0000","last_undegraded":"2024-04-24T16:00:20.357003+0000","last_fullsized":"2024-04-24T16:00:20.357003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":208,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835853+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T16:00:20.835853+0000","last_peered":"2024-04-24T16:00:20.835853+0000","last_clean":"2024-04-24T16:00:20.835853+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T16:00:20.835853+0000","last_undegraded":"2024-04-24T16:00:20.835853+0000","last_fullsized":"2024-04-24T16:00:20.835853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"59'44","reported_seq":282,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357572+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:20.357572+0000","last_peered":"2024-04-24T16:00:20.357572+0000","last_clean":"2024-04-24T16:00:20.357572+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:20.357572+0000","last_undegraded":"2024-04-24T16:00:20.357572+0000","last_fullsized":"2024-04-24T16:00:20.357572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":282,"num_read_kb":10860,"num_write":55,"num_write_kb":8225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835872+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:20.835872+0000","last_peered":"2024-04-24T16:00:20.835872+0000","last_clean":"2024-04-24T16:00:20.835872+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:20.835872+0000","last_undegraded":"2024-04-24T16:00:20.835872+0000","last_fullsized":"2024-04-24T16:00:20.835872+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":119,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"62'255","reported_seq":886,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358283+0000","last_change":"2024-04-24T16:00:21.358613+0000","last_active":"2024-04-24T16:00:21.358242+0000","last_peered":"2024-04-24T16:00:20.356984+0000","last_clean":"2024-04-24T16:00:20.356984+0000","last_became_active":"2024-04-24T15:59:28.190693+0000","last_became_peered":"2024-04-24T15:59:28.190693+0000","last_unstale":"2024-04-24T16:00:21.358283+0000","last_undegraded":"2024-04-24T16:00:21.358283+0000","last_fullsized":"2024-04-24T16:00:21.358283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:40.923211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":25165862,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1519,"num_read_kb":46466,"num_write":254,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835838+0000","last_change":"2024-04-24T15:59:52.896635+0000","last_active":"2024-04-24T16:00:20.835838+0000","last_peered":"2024-04-24T16:00:20.835838+0000","last_clean":"2024-04-24T16:00:20.835838+0000","last_became_active":"2024-04-24T15:59:50.311714+0000","last_became_peered":"2024-04-24T15:59:50.311714+0000","last_unstale":"2024-04-24T16:00:20.835838+0000","last_undegraded":"2024-04-24T16:00:20.835838+0000","last_fullsized":"2024-04-24T16:00:20.835838+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:57.999783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037330000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":74,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835859+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:20.835859+0000","last_peered":"2024-04-24T16:00:20.835859+0000","last_clean":"2024-04-24T16:00:20.835859+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:20.835859+0000","last_undegraded":"2024-04-24T16:00:20.835859+0000","last_fullsized":"2024-04-24T16:00:20.835859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"58'142","reported_seq":631,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357101+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:20.357101+0000","last_peered":"2024-04-24T16:00:20.357101+0000","last_clean":"2024-04-24T16:00:20.357101+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:20.357101+0000","last_undegraded":"2024-04-24T16:00:20.357101+0000","last_fullsized":"2024-04-24T16:00:20.357101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:03.811282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":25165881,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":432,"num_read_kb":57544,"num_write":131,"num_write_kb":49028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":36,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358399+0000","last_change":"2024-04-24T16:00:21.358878+0000","last_active":"2024-04-24T16:00:21.358366+0000","last_peered":"2024-04-24T16:00:20.840554+0000","last_clean":"2024-04-24T16:00:20.840554+0000","last_became_active":"2024-04-24T15:59:50.269342+0000","last_became_peered":"2024-04-24T15:59:50.269342+0000","last_unstale":"2024-04-24T16:00:21.358399+0000","last_undegraded":"2024-04-24T16:00:21.358399+0000","last_fullsized":"2024-04-24T16:00:21.358399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:00:28.126370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.837324+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:20.837324+0000","last_peered":"2024-04-24T16:00:20.837324+0000","last_clean":"2024-04-24T16:00:20.837324+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:20.837324+0000","last_undegraded":"2024-04-24T16:00:20.837324+0000","last_fullsized":"2024-04-24T16:00:20.837324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'101","reported_seq":808,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:41.233351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":20971589,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":835,"num_read_kb":43845,"num_write":75,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":34,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.922604+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:20.922604+0000","last_peered":"2024-04-24T16:00:20.922604+0000","last_clean":"2024-04-24T16:00:20.922604+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:20.922604+0000","last_undegraded":"2024-04-24T16:00:20.922604+0000","last_fullsized":"2024-04-24T16:00:20.922604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"58'4","reported_seq":73,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835873+0000","last_change":"2024-04-24T15:59:48.261056+0000","last_active":"2024-04-24T16:00:20.835873+0000","last_peered":"2024-04-24T16:00:20.835873+0000","last_clean":"2024-04-24T16:00:20.835873+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T16:00:20.835873+0000","last_undegraded":"2024-04-24T16:00:20.835873+0000","last_fullsized":"2024-04-24T16:00:20.835873+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"59'8","reported_seq":95,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.360284+0000","last_change":"2024-04-24T16:00:21.360552+0000","last_active":"2024-04-24T16:00:21.360253+0000","last_peered":"2024-04-24T16:00:20.837376+0000","last_clean":"2024-04-24T16:00:20.837376+0000","last_became_active":"2024-04-24T15:59:28.271464+0000","last_became_peered":"2024-04-24T15:59:28.271464+0000","last_unstale":"2024-04-24T16:00:21.360284+0000","last_undegraded":"2024-04-24T16:00:21.360284+0000","last_fullsized":"2024-04-24T16:00:21.360284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:42.551625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165877,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":37,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.931278+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:20.931278+0000","last_peered":"2024-04-24T16:00:20.931278+0000","last_clean":"2024-04-24T16:00:20.931278+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:20.931278+0000","last_undegraded":"2024-04-24T16:00:20.931278+0000","last_fullsized":"2024-04-24T16:00:20.931278+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":83886282,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":3068,"num_read_kb":158715,"num_write":515,"num_write_kb":131119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167829504,"data_stored":167772564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":542,"ondisk_log_size":542,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217895,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":111,"num_read_kb":4807,"num_write":43,"num_write_kb":131210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640279,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375932,"kb_used_data":78572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93995908,"statfs":{"total":96636764160,"available":96251809792,"internally_reserved":0,"allocated":80457728,"data_stored":79989123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.435}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.308}]}]},{"osd":4,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396424,"kb_used_data":99060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93975416,"statfs":{"total":96636764160,"available":96230825984,"internally_reserved":0,"allocated":101437440,"data_stored":100966082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.379},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.33800000000000002}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28199999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372420,"kb_used_data":75060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999420,"statfs":{"total":96636764160,"available":96255406080,"internally_reserved":0,"allocated":76861440,"data_stored":76390607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54900000000000004}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32000000000000001}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":737280,"data_stored":297164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.31},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26100000000000001}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32800000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.499}]}]},{"osd":0,"up_from":11,"seq":47244640280,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388216,"kb_used_data":86764,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93983624,"statfs":{"total":96636764160,"available":96239230976,"internally_reserved":0,"allocated":88846336,"data_stored":88377747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.317},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.503},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.43099999999999999}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35399999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392904,"kb_used_data":91444,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93978936,"statfs":{"total":96636764160,"available":96234430464,"internally_reserved":0,"allocated":93638656,"data_stored":93167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.435}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48099999999999998}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54534144,"data_stored":54526028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75513856,"data_stored":75497601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:24.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:24.958 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:24.961+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:25.006 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:25.006 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:25.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.701421508999999, 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.010 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.691801445000003, 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.011 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.671400878, 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:25.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:25.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.658469076000003, 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.658435324999999, 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:25.025 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:25.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:25.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.656912914999999, 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:25.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.651501799000002, 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.036 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:25.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.643753281000002, 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:25.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:25.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.641623258000003, 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:25.045 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:25.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:25.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.625920301000001, 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.053 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.625892313999998, 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:25.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:25.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.609121238, 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:25.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:25.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:25.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.603777714000003, 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.065 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:25.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:25.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.591567468999997, 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:25.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:25.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:25.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:25.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.584132304000001, 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:25.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:25.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.082 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:57 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.583186126999998, 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038713490000000003, 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424098+0000", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424097+0000", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424102+0000", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3149999999999999e-06 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424118+0000", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5625999999999999e-05 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424125+0000", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2960000000000001e-06 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424490+0000", 2024-04-24T16:00:25.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000364781 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424569+0000", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8584000000000005e-05 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.424869+0000", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030080800000000001 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430343+0000", 2024-04-24T16:00:25.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054732189999999997 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462762+0000", 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032419199000000003 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462789+0000", 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7463000000000001e-05 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462811+0000", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2048999999999998e-05 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:59 3.2 3:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.535985932000003, 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039387966000000003, 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:25.088 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 59 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471298+0000", 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.471297+0000", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479102+0000", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078047949999999998 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479124+0000", 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2449000000000001e-05 2024-04-24T16:00:25.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479132+0000", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6190000000000002e-06 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.479193+0000", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1668999999999995e-05 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510682+0000", 2024-04-24T16:00:25.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031488457999999997 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.510686+0000", 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2150000000000003e-06 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.399876405000001, 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:25.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.096 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.395640477999997, 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:25.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:25.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.388800711999998, 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:25.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:25.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:25.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:25.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.108 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:25.108 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:25.108 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:25.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:25.111 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:25.111+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:25.210 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:25.212+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:25.277 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:25.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-04-24T16:00:24.372478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218694615,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":7,"num_read":3333,"num_read_kb":163652,"num_write":723,"num_write_kb":264751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":2223976,"kb_used_data":431620,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564007064,"statfs":{"total":579820584960,"available":577543233536,"internally_reserved":0,"allocated":441978880,"data_stored":439188531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":296,"num_read_kb":3255,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016107"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":86,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358229+0000","last_change":"2024-04-24T16:00:21.358408+0000","last_active":"2024-04-24T16:00:21.358210+0000","last_peered":"2024-04-24T16:00:21.053511+0000","last_clean":"2024-04-24T16:00:21.053511+0000","last_became_active":"2024-04-24T15:59:28.190640+0000","last_became_peered":"2024-04-24T15:59:28.190640+0000","last_unstale":"2024-04-24T16:00:21.358229+0000","last_undegraded":"2024-04-24T16:00:21.358229+0000","last_fullsized":"2024-04-24T16:00:21.358229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:50:16.740896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357547+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:20.357547+0000","last_peered":"2024-04-24T16:00:20.357547+0000","last_clean":"2024-04-24T16:00:20.357547+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:20.357547+0000","last_undegraded":"2024-04-24T16:00:20.357547+0000","last_fullsized":"2024-04-24T16:00:20.357547+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":90,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357208+0000","last_change":"2024-04-24T15:59:48.255715+0000","last_active":"2024-04-24T16:00:20.357208+0000","last_peered":"2024-04-24T16:00:20.357208+0000","last_clean":"2024-04-24T16:00:20.357208+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T16:00:20.357208+0000","last_undegraded":"2024-04-24T16:00:20.357208+0000","last_fullsized":"2024-04-24T16:00:20.357208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053524+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:21.053524+0000","last_peered":"2024-04-24T16:00:21.053524+0000","last_clean":"2024-04-24T16:00:21.053524+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:21.053524+0000","last_undegraded":"2024-04-24T16:00:21.053524+0000","last_fullsized":"2024-04-24T16:00:21.053524+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":43,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.082668+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:21.082668+0000","last_peered":"2024-04-24T16:00:21.082668+0000","last_clean":"2024-04-24T16:00:21.082668+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:21.082668+0000","last_undegraded":"2024-04-24T16:00:21.082668+0000","last_fullsized":"2024-04-24T16:00:21.082668+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":86,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357003+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:20.357003+0000","last_peered":"2024-04-24T16:00:20.357003+0000","last_clean":"2024-04-24T16:00:20.357003+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:20.357003+0000","last_undegraded":"2024-04-24T16:00:20.357003+0000","last_fullsized":"2024-04-24T16:00:20.357003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":208,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835853+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T16:00:20.835853+0000","last_peered":"2024-04-24T16:00:20.835853+0000","last_clean":"2024-04-24T16:00:20.835853+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T16:00:20.835853+0000","last_undegraded":"2024-04-24T16:00:20.835853+0000","last_fullsized":"2024-04-24T16:00:20.835853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"59'44","reported_seq":282,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357572+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:20.357572+0000","last_peered":"2024-04-24T16:00:20.357572+0000","last_clean":"2024-04-24T16:00:20.357572+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:20.357572+0000","last_undegraded":"2024-04-24T16:00:20.357572+0000","last_fullsized":"2024-04-24T16:00:20.357572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":282,"num_read_kb":10860,"num_write":55,"num_write_kb":8225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835872+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:20.835872+0000","last_peered":"2024-04-24T16:00:20.835872+0000","last_clean":"2024-04-24T16:00:20.835872+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:20.835872+0000","last_undegraded":"2024-04-24T16:00:20.835872+0000","last_fullsized":"2024-04-24T16:00:20.835872+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":119,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"62'255","reported_seq":886,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358283+0000","last_change":"2024-04-24T16:00:21.358613+0000","last_active":"2024-04-24T16:00:21.358242+0000","last_peered":"2024-04-24T16:00:20.356984+0000","last_clean":"2024-04-24T16:00:20.356984+0000","last_became_active":"2024-04-24T15:59:28.190693+0000","last_became_peered":"2024-04-24T15:59:28.190693+0000","last_unstale":"2024-04-24T16:00:21.358283+0000","last_undegraded":"2024-04-24T16:00:21.358283+0000","last_fullsized":"2024-04-24T16:00:21.358283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:40.923211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":25165862,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1519,"num_read_kb":46466,"num_write":254,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835838+0000","last_change":"2024-04-24T15:59:52.896635+0000","last_active":"2024-04-24T16:00:20.835838+0000","last_peered":"2024-04-24T16:00:20.835838+0000","last_clean":"2024-04-24T16:00:20.835838+0000","last_became_active":"2024-04-24T15:59:50.311714+0000","last_became_peered":"2024-04-24T15:59:50.311714+0000","last_unstale":"2024-04-24T16:00:20.835838+0000","last_undegraded":"2024-04-24T16:00:20.835838+0000","last_fullsized":"2024-04-24T16:00:20.835838+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:57.999783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037330000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":74,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835859+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:20.835859+0000","last_peered":"2024-04-24T16:00:20.835859+0000","last_clean":"2024-04-24T16:00:20.835859+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:20.835859+0000","last_undegraded":"2024-04-24T16:00:20.835859+0000","last_fullsized":"2024-04-24T16:00:20.835859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"58'142","reported_seq":631,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357101+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:20.357101+0000","last_peered":"2024-04-24T16:00:20.357101+0000","last_clean":"2024-04-24T16:00:20.357101+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:20.357101+0000","last_undegraded":"2024-04-24T16:00:20.357101+0000","last_fullsized":"2024-04-24T16:00:20.357101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:03.811282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":25165881,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":432,"num_read_kb":57544,"num_write":131,"num_write_kb":49028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":36,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358399+0000","last_change":"2024-04-24T16:00:21.358878+0000","last_active":"2024-04-24T16:00:21.358366+0000","last_peered":"2024-04-24T16:00:20.840554+0000","last_clean":"2024-04-24T16:00:20.840554+0000","last_became_active":"2024-04-24T15:59:50.269342+0000","last_became_peered":"2024-04-24T15:59:50.269342+0000","last_unstale":"2024-04-24T16:00:21.358399+0000","last_undegraded":"2024-04-24T16:00:21.358399+0000","last_fullsized":"2024-04-24T16:00:21.358399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:00:28.126370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.837324+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:20.837324+0000","last_peered":"2024-04-24T16:00:20.837324+0000","last_clean":"2024-04-24T16:00:20.837324+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:20.837324+0000","last_undegraded":"2024-04-24T16:00:20.837324+0000","last_fullsized":"2024-04-24T16:00:20.837324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'101","reported_seq":808,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:41.233351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":20971589,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":835,"num_read_kb":43845,"num_write":75,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":34,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.922604+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:20.922604+0000","last_peered":"2024-04-24T16:00:20.922604+0000","last_clean":"2024-04-24T16:00:20.922604+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:20.922604+0000","last_undegraded":"2024-04-24T16:00:20.922604+0000","last_fullsized":"2024-04-24T16:00:20.922604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"58'4","reported_seq":73,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835873+0000","last_change":"2024-04-24T15:59:48.261056+0000","last_active":"2024-04-24T16:00:20.835873+0000","last_peered":"2024-04-24T16:00:20.835873+0000","last_clean":"2024-04-24T16:00:20.835873+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T16:00:20.835873+0000","last_undegraded":"2024-04-24T16:00:20.835873+0000","last_fullsized":"2024-04-24T16:00:20.835873+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"59'8","reported_seq":95,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.360284+0000","last_change":"2024-04-24T16:00:21.360552+0000","last_active":"2024-04-24T16:00:21.360253+0000","last_peered":"2024-04-24T16:00:20.837376+0000","last_clean":"2024-04-24T16:00:20.837376+0000","last_became_active":"2024-04-24T15:59:28.271464+0000","last_became_peered":"2024-04-24T15:59:28.271464+0000","last_unstale":"2024-04-24T16:00:21.360284+0000","last_undegraded":"2024-04-24T16:00:21.360284+0000","last_fullsized":"2024-04-24T16:00:21.360284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:42.551625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165877,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":37,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.931278+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:20.931278+0000","last_peered":"2024-04-24T16:00:20.931278+0000","last_clean":"2024-04-24T16:00:20.931278+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:20.931278+0000","last_undegraded":"2024-04-24T16:00:20.931278+0000","last_fullsized":"2024-04-24T16:00:20.931278+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":83886282,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":3068,"num_read_kb":158715,"num_write":515,"num_write_kb":131119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167829504,"data_stored":167772564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":542,"ondisk_log_size":542,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217895,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":111,"num_read_kb":4807,"num_write":43,"num_write_kb":131210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640279,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375932,"kb_used_data":78572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93995908,"statfs":{"total":96636764160,"available":96251809792,"internally_reserved":0,"allocated":80457728,"data_stored":79989123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.435}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.308}]}]},{"osd":4,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396424,"kb_used_data":99060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93975416,"statfs":{"total":96636764160,"available":96230825984,"internally_reserved":0,"allocated":101437440,"data_stored":100966082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.379},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.33800000000000002}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28199999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372420,"kb_used_data":75060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999420,"statfs":{"total":96636764160,"available":96255406080,"internally_reserved":0,"allocated":76861440,"data_stored":76390607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54900000000000004}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32000000000000001}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":737280,"data_stored":297164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.31},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26100000000000001}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32800000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.499}]}]},{"osd":0,"up_from":11,"seq":47244640280,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388216,"kb_used_data":86764,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93983624,"statfs":{"total":96636764160,"available":96239230976,"internally_reserved":0,"allocated":88846336,"data_stored":88377747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.317},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.503},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.43099999999999999}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35399999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392904,"kb_used_data":91444,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93978936,"statfs":{"total":96636764160,"available":96234430464,"internally_reserved":0,"allocated":93638656,"data_stored":93167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.435}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48099999999999998}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54534144,"data_stored":54526028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75513856,"data_stored":75497601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:25.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T16:00:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:25.294 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:25.294 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:25.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:25.346 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:25.348+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:25.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:00:25.431 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:25.433+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:25.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:25.501 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:25.502 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:25.502 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:25.502 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:25.502 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:25.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:25.513 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:25.516+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:25.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:25.529 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:25.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:25.580 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:25.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:25.615 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:25.617+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:25.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:25.730 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:25.733+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:25.819 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:25.821+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:25.820 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":115,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":68,"num_osds":6,"num_up_osds":6,"osd_up_since":1713974321,"num_in_osds":6,"osd_in_since":1713974312,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"peering","count":4}],"num_pgs":21,"num_pools":4,"num_objects":88,"data_bytes":218694615,"bytes_used":2277351424,"bytes_avail":577543233536,"bytes_total":579820584960,"inactive_pgs_ratio":0.190476194024086,"read_bytes_sec":833151,"write_bytes_sec":1023,"read_op_per_sec":73,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-04-24T15:59:43.763015+0000","services":{}},"progress_events":{}} 2024-04-24T16:00:25.823 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:25.823 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:25.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:25.835 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-24T16:00:25.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:25.919 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:25.921+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:25.998 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:25.998 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:26.020 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:26.022+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:26.120 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:26.123+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:26.131 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.131 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:26.185 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:26.200 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-24T16:00:25.770015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218694615,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":7,"num_read":3333,"num_read_kb":163652,"num_write":723,"num_write_kb":264751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":2223976,"kb_used_data":431620,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564007064,"statfs":{"total":579820584960,"available":577543233536,"internally_reserved":0,"allocated":441978880,"data_stored":439188531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":296,"num_read_kb":3255,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000661"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":86,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358229+0000","last_change":"2024-04-24T16:00:21.358408+0000","last_active":"2024-04-24T16:00:21.358210+0000","last_peered":"2024-04-24T16:00:21.053511+0000","last_clean":"2024-04-24T16:00:21.053511+0000","last_became_active":"2024-04-24T15:59:28.190640+0000","last_became_peered":"2024-04-24T15:59:28.190640+0000","last_unstale":"2024-04-24T16:00:21.358229+0000","last_undegraded":"2024-04-24T16:00:21.358229+0000","last_fullsized":"2024-04-24T16:00:21.358229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:50:16.740896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357547+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:20.357547+0000","last_peered":"2024-04-24T16:00:20.357547+0000","last_clean":"2024-04-24T16:00:20.357547+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:20.357547+0000","last_undegraded":"2024-04-24T16:00:20.357547+0000","last_fullsized":"2024-04-24T16:00:20.357547+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":90,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357208+0000","last_change":"2024-04-24T15:59:48.255715+0000","last_active":"2024-04-24T16:00:20.357208+0000","last_peered":"2024-04-24T16:00:20.357208+0000","last_clean":"2024-04-24T16:00:20.357208+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T16:00:20.357208+0000","last_undegraded":"2024-04-24T16:00:20.357208+0000","last_fullsized":"2024-04-24T16:00:20.357208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053524+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:21.053524+0000","last_peered":"2024-04-24T16:00:21.053524+0000","last_clean":"2024-04-24T16:00:21.053524+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:21.053524+0000","last_undegraded":"2024-04-24T16:00:21.053524+0000","last_fullsized":"2024-04-24T16:00:21.053524+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":43,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.082668+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:21.082668+0000","last_peered":"2024-04-24T16:00:21.082668+0000","last_clean":"2024-04-24T16:00:21.082668+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:21.082668+0000","last_undegraded":"2024-04-24T16:00:21.082668+0000","last_fullsized":"2024-04-24T16:00:21.082668+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":86,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357003+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:20.357003+0000","last_peered":"2024-04-24T16:00:20.357003+0000","last_clean":"2024-04-24T16:00:20.357003+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:20.357003+0000","last_undegraded":"2024-04-24T16:00:20.357003+0000","last_fullsized":"2024-04-24T16:00:20.357003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":208,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835853+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T16:00:20.835853+0000","last_peered":"2024-04-24T16:00:20.835853+0000","last_clean":"2024-04-24T16:00:20.835853+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T16:00:20.835853+0000","last_undegraded":"2024-04-24T16:00:20.835853+0000","last_fullsized":"2024-04-24T16:00:20.835853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"59'44","reported_seq":282,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357572+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:20.357572+0000","last_peered":"2024-04-24T16:00:20.357572+0000","last_clean":"2024-04-24T16:00:20.357572+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:20.357572+0000","last_undegraded":"2024-04-24T16:00:20.357572+0000","last_fullsized":"2024-04-24T16:00:20.357572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":282,"num_read_kb":10860,"num_write":55,"num_write_kb":8225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835872+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:20.835872+0000","last_peered":"2024-04-24T16:00:20.835872+0000","last_clean":"2024-04-24T16:00:20.835872+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:20.835872+0000","last_undegraded":"2024-04-24T16:00:20.835872+0000","last_fullsized":"2024-04-24T16:00:20.835872+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":119,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"62'255","reported_seq":886,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358283+0000","last_change":"2024-04-24T16:00:21.358613+0000","last_active":"2024-04-24T16:00:21.358242+0000","last_peered":"2024-04-24T16:00:20.356984+0000","last_clean":"2024-04-24T16:00:20.356984+0000","last_became_active":"2024-04-24T15:59:28.190693+0000","last_became_peered":"2024-04-24T15:59:28.190693+0000","last_unstale":"2024-04-24T16:00:21.358283+0000","last_undegraded":"2024-04-24T16:00:21.358283+0000","last_fullsized":"2024-04-24T16:00:21.358283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:40.923211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":25165862,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1519,"num_read_kb":46466,"num_write":254,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835838+0000","last_change":"2024-04-24T15:59:52.896635+0000","last_active":"2024-04-24T16:00:20.835838+0000","last_peered":"2024-04-24T16:00:20.835838+0000","last_clean":"2024-04-24T16:00:20.835838+0000","last_became_active":"2024-04-24T15:59:50.311714+0000","last_became_peered":"2024-04-24T15:59:50.311714+0000","last_unstale":"2024-04-24T16:00:20.835838+0000","last_undegraded":"2024-04-24T16:00:20.835838+0000","last_fullsized":"2024-04-24T16:00:20.835838+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:57.999783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037330000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":74,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835859+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:20.835859+0000","last_peered":"2024-04-24T16:00:20.835859+0000","last_clean":"2024-04-24T16:00:20.835859+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:20.835859+0000","last_undegraded":"2024-04-24T16:00:20.835859+0000","last_fullsized":"2024-04-24T16:00:20.835859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"58'142","reported_seq":631,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.357101+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:20.357101+0000","last_peered":"2024-04-24T16:00:20.357101+0000","last_clean":"2024-04-24T16:00:20.357101+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:20.357101+0000","last_undegraded":"2024-04-24T16:00:20.357101+0000","last_fullsized":"2024-04-24T16:00:20.357101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:03.811282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":25165881,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":432,"num_read_kb":57544,"num_write":131,"num_write_kb":49028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":36,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.358399+0000","last_change":"2024-04-24T16:00:21.358878+0000","last_active":"2024-04-24T16:00:21.358366+0000","last_peered":"2024-04-24T16:00:20.840554+0000","last_clean":"2024-04-24T16:00:20.840554+0000","last_became_active":"2024-04-24T15:59:50.269342+0000","last_became_peered":"2024-04-24T15:59:50.269342+0000","last_unstale":"2024-04-24T16:00:21.358399+0000","last_undegraded":"2024-04-24T16:00:21.358399+0000","last_fullsized":"2024-04-24T16:00:21.358399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:00:28.126370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.837324+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:20.837324+0000","last_peered":"2024-04-24T16:00:20.837324+0000","last_clean":"2024-04-24T16:00:20.837324+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:20.837324+0000","last_undegraded":"2024-04-24T16:00:20.837324+0000","last_fullsized":"2024-04-24T16:00:20.837324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'101","reported_seq":808,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:21.053502+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:21.053502+0000","last_peered":"2024-04-24T16:00:21.053502+0000","last_clean":"2024-04-24T16:00:21.053502+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:21.053502+0000","last_undegraded":"2024-04-24T16:00:21.053502+0000","last_fullsized":"2024-04-24T16:00:21.053502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:41.233351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":20971589,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":835,"num_read_kb":43845,"num_write":75,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":34,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.922604+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:20.922604+0000","last_peered":"2024-04-24T16:00:20.922604+0000","last_clean":"2024-04-24T16:00:20.922604+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:20.922604+0000","last_undegraded":"2024-04-24T16:00:20.922604+0000","last_fullsized":"2024-04-24T16:00:20.922604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"58'4","reported_seq":73,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.835873+0000","last_change":"2024-04-24T15:59:48.261056+0000","last_active":"2024-04-24T16:00:20.835873+0000","last_peered":"2024-04-24T16:00:20.835873+0000","last_clean":"2024-04-24T16:00:20.835873+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T16:00:20.835873+0000","last_undegraded":"2024-04-24T16:00:20.835873+0000","last_fullsized":"2024-04-24T16:00:20.835873+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"59'8","reported_seq":95,"reported_epoch":64,"state":"peering","last_fresh":"2024-04-24T16:00:21.360284+0000","last_change":"2024-04-24T16:00:21.360552+0000","last_active":"2024-04-24T16:00:21.360253+0000","last_peered":"2024-04-24T16:00:20.837376+0000","last_clean":"2024-04-24T16:00:20.837376+0000","last_became_active":"2024-04-24T15:59:28.271464+0000","last_became_peered":"2024-04-24T15:59:28.271464+0000","last_unstale":"2024-04-24T16:00:21.360284+0000","last_undegraded":"2024-04-24T16:00:21.360284+0000","last_fullsized":"2024-04-24T16:00:21.360284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:42.551625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165877,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":37,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-24T16:00:20.931278+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:20.931278+0000","last_peered":"2024-04-24T16:00:20.931278+0000","last_clean":"2024-04-24T16:00:20.931278+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:20.931278+0000","last_undegraded":"2024-04-24T16:00:20.931278+0000","last_fullsized":"2024-04-24T16:00:20.931278+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":83886282,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":3068,"num_read_kb":158715,"num_write":515,"num_write_kb":131119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":39,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167829504,"data_stored":167772564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":542,"ondisk_log_size":542,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217895,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":111,"num_read_kb":4807,"num_write":43,"num_write_kb":131210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640279,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375932,"kb_used_data":78572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93995908,"statfs":{"total":96636764160,"available":96251809792,"internally_reserved":0,"allocated":80457728,"data_stored":79989123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.435}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.308}]}]},{"osd":4,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396424,"kb_used_data":99060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93975416,"statfs":{"total":96636764160,"available":96230825984,"internally_reserved":0,"allocated":101437440,"data_stored":100966082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.379},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.33800000000000002}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28199999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372420,"kb_used_data":75060,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999420,"statfs":{"total":96636764160,"available":96255406080,"internally_reserved":0,"allocated":76861440,"data_stored":76390607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54900000000000004}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32000000000000001}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":737280,"data_stored":297164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.31},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26100000000000001}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32800000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45700000000000002}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.499}]}]},{"osd":0,"up_from":11,"seq":47244640280,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388216,"kb_used_data":86764,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93983624,"statfs":{"total":96636764160,"available":96239230976,"internally_reserved":0,"allocated":88846336,"data_stored":88377747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.317},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.41699999999999998}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.503},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.43099999999999999}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.35399999999999998}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392904,"kb_used_data":91444,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93978936,"statfs":{"total":96636764160,"available":96234430464,"internally_reserved":0,"allocated":93638656,"data_stored":93167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.435}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48099999999999998}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54534144,"data_stored":54526028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75513856,"data_stored":75497601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58720401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:26.226 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:26.229+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:26.306 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.306 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:26.356 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:26.359+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:00:26.423 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:26.426+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:26.482 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.482 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:26.533 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:26.536+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:26.626 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:26.627+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:26.655 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.656 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:26.726 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:26.729+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:26.789 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.789 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:26.842 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:26.845+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:26.928 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:26.931+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:26.965 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:26.965 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:26.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:27.059 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:27.062+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:27.130 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:27.133+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:27.139 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:27.139 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:27.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:27.231 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:27.233+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:27.300 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:27.300 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:27.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:27.332 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:27.333+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:27.433 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:27.435+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:27.493 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:27.494 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:27.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:27.534 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:27.536+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:27.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:27.634 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:27.636+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:27.645 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:27.645 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:27.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:27.745 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:27.747+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:27.773 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:27.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:27.813 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:27.834 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:27.837 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:27.838+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:27.842 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:27.842 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:27.842 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:27.842 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:27.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:27.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:27.874 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:27.876 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:27.876 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:27.937 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:27.939+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:28.012 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:28.012 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:28.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:28.037 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:28.040+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:28.138 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:28.140+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:28.206 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.901381600000001, 2024-04-24T16:00:28.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:28.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.891761535999997, 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.212 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:28.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:28.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.871360969000001, 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:28.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:28.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:28.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:28.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.858429166999997, 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:28.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.858395416, 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:28.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:28.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:28.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:28.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.230 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.856873006000001, 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:28.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.851461890000003, 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.843713372000003, 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:28.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.841583348999997, 2024-04-24T16:00:28.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:28.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:28.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.825880392000002, 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:28.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:28.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.825852404999999, 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:28.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.809081329000001, 2024-04-24T16:00:28.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.803737804999997, 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.263 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:28.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.791527559999999, 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:28.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:28.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.784092395000002, 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.278 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:28.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:28.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:28.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.599836496000002, 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:28.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.595600568999998, 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:28.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.588760803, 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:28.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:28.297 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8347043850000002, 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:28.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8324797209999999, 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.303 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:00:28.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:00:28.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:00:28.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.310 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.311 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:28.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:28.313 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:28.314+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:28.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:00:28.414 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:28.415+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:28.503 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:28.503 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:28.503 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:28.503 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:28.514 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:28.516+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:28.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:28.616 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:28.618+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:28.695 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:28.695 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:28.696 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:28.696 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:28.696 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:28.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:28.750 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:28.752+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:28.817 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:28.818+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4712399920000001, 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.470376645, 2024-04-24T16:00:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.46699438, 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4631736990000004, 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014029229999999999, 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387204+0000", 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387733+0000", 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052950899999999995 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387753+0000", 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9383000000000001e-05 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387762+0000", 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6409999999999997e-06 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387826+0000", 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3915e-05 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387833+0000", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-06 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.388608+0000", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077458099999999999 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4598346939999995, 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480992, 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390543+0000", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391057+0000", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00051421400000000003 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391073+0000", 2024-04-24T16:00:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519000000000002e-05 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391081+0000", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1980000000000001e-06 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391126+0000", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5429000000000002e-05 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391132+0000", 2024-04-24T16:00:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3770000000000004e-06 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.392025+0000", 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089311900000000005 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head, version: 56'185, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head@56'185, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4457905999999996, 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001542362, 2024-04-24T16:00:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404587+0000", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405173+0000", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000586328 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405189+0000", 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5630000000000001e-05 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405196+0000", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2930000000000003e-06 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405249+0000", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2280000000000001e-05 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405256+0000", 2024-04-24T16:00:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7200000000000006e-06 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.406130+0000", 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087399199999999998 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4416487199999999, 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013967070000000001, 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408729+0000", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409290+0000", 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056137799999999996 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409314+0000", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3377999999999999e-05 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409323+0000", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3130000000000004e-06 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409384+0000", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0752999999999999e-05 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409389+0000", 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5400000000000003e-06 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.410127+0000", 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073712699999999996 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.438280024, 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001435471, 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412098+0000", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412702+0000", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060413800000000005 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412717+0000", 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5545000000000001e-05 2024-04-24T16:00:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412725+0000", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7060000000000004e-06 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412782+0000", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6666000000000003e-05 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412787+0000", 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5770000000000001e-06 2024-04-24T16:00:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.413534+0000", 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074670499999999996 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4312327109999998, 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015370780000000001, 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419145+0000", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419734+0000", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058931100000000002 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419750+0000", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5996000000000001e-05 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419758+0000", 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9629999999999998e-06 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419807+0000", 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8251999999999998e-05 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419813+0000", 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6590000000000001e-06 2024-04-24T16:00:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.420683+0000", 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086977599999999995 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4203708949999996, 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001420075, 2024-04-24T16:00:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430007+0000", 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430585+0000", 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057783400000000005 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430603+0000", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8485e-05 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430612+0000", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6640000000000004e-06 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430679+0000", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7578999999999998e-05 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430686+0000", 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0190000000000003e-06 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.431428+0000", 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074226900000000005 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4167990420000001, 2024-04-24T16:00:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.411765613, 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:00:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:00:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4078108650000001, 2024-04-24T16:00:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.4037045460000002, 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001428697, 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446673+0000", 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447203+0000", 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053015899999999999 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447226+0000", 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2578e-05 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447235+0000", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8300000000000002e-06 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447305+0000", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0418000000000002e-05 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447325+0000", 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9647999999999999e-05 2024-04-24T16:00:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.448103+0000", 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077794699999999997 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3998913259999997, 2024-04-24T16:00:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001591912, 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450486+0000", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451016+0000", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052945499999999996 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451033+0000", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7416000000000001e-05 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451042+0000", 2024-04-24T16:00:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1759999999999999e-06 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451098+0000", 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.609e-05 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451106+0000", 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9230000000000002e-06 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.452079+0000", 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097278800000000004 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3939916859999997, 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016296419999999999, 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456386+0000", 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457118+0000", 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073264499999999998 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457144+0000", 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5667999999999999e-05 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457158+0000", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4542000000000001e-05 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457220+0000", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1935999999999998e-05 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457227+0000", 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2110000000000003e-06 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.458017+0000", 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078992899999999998 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3900775730000001, 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001562031, 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460300+0000", 2024-04-24T16:00:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460929+0000", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062916800000000004 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460947+0000", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7985000000000001e-05 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460958+0000", 2024-04-24T16:00:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1345000000000001e-05 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461024+0000", 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5708000000000002e-05 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461032+0000", 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.661e-06 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461863+0000", 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083151199999999996 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3864252620000004, 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001413178, 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463952+0000", 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464492+0000", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053963000000000004 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464509+0000", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7472000000000001e-05 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464521+0000", 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1747e-05 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464582+0000", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0881999999999997e-05 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464590+0000", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2269999999999993e-06 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.465367+0000", 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077665299999999998 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3828806980000001, 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:00:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:00:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.4677558049999999, 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:00:28.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:28.930 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:28.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:28.933 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:28.934+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:28.956 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T16:00:28.961 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:28.962 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:28.962 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:28.963 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:28.963 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:28.965 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:28.965 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:28.966 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:28.990 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:28.990 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:28.990 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119473 s, 87.8 MB/s 2024-04-24T16:00:28.992 INFO:tasks.workunit.client.0.smithi122.stderr:++ cat /tmp/tmp.P3d5b6KQ4E/quiesce 2024-04-24T16:00:28.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-04-24T16:00:28.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ cat /tmp/tmp.P3d5b6KQ4E/unquiesce 2024-04-24T16:00:28.994 INFO:tasks.workunit.client.0.smithi122.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-04-24T16:00:28.994 INFO:tasks.workunit.client.0.smithi122.stderr:+ touch /tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:28.995 INFO:tasks.workunit.client.0.smithi122.stderr:+ chmod +x /tmp/tmp.P3d5b6KQ4E/quiesce.sh 2024-04-24T16:00:28.996 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-04-24T16:00:28.997 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd snap create rbd/testrbdnbd37497@quiesce2 2024-04-24T16:00:28.997 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@quiesce2 2024-04-24T16:00:29.032 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:29.034+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:29.048 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-04-24T16:00:29.048 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-04-24T16:00:29.053 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:29.053 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:29.053 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:29.053 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:29.054 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:29.054 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:29.057 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:29.057 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:29.057 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:29.072 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.072 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:29.073 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:29.073 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:29.076 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:29.076 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:29.076 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759882 s, 138 MB/s 2024-04-24T16:00:29.078 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-04-24T16:00:29.079 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@quiesce2 2024-04-24T16:00:29.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:29.133 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:29.135+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:29.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:29.233 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:29.235+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:29.248 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:29.248 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:29.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:29.334 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:29.336+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:29.401 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.116589732999998, 2024-04-24T16:00:29.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:29.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:29.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:29.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.108623702999999, 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.108589508000001, 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.108570870000001, 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.101248132999999, 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.079589110000001, 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:29.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:29.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.075340476000001, 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.460 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.075305958000001, 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:29.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.464 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.045144854, 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.994049914999998, 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.940531403000001, 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.817191243000003, 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.800321787999998, 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.792105362999997, 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.489 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.778565215, 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:29.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.776680546000001, 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:29.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.756072287000002, 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:29.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:29.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:29.503 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.746150786999998, 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:29.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.507 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.741502529000002, 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.511 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 40.740008738, 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:29.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.515 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.516 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.516 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:29.517 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:29.519+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:29.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:29.562 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:29.576 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-24T16:00:29.398279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":219743267,"num_objects":91,"num_object_clones":1,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":7,"num_read":3606,"num_read_kb":165885,"num_write":743,"num_write_kb":265781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":62914724,"num_keys_recovered":14,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":740,"ondisk_log_size":740,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2230484,"kb_used_data":434044,"kb_used_omap":9,"kb_used_meta":1796342,"kb_avail":564000556,"statfs":{"total":579820584960,"available":577536569344,"internally_reserved":0,"allocated":444461056,"data_stored":447743245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1839454224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1048652,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":273,"num_read_kb":2233,"num_write":20,"num_write_kb":1030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":62914724,"num_keys_recovered":14,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.628624"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":108,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.380156+0000","last_change":"2024-04-24T16:00:22.519701+0000","last_active":"2024-04-24T16:00:25.380156+0000","last_peered":"2024-04-24T16:00:25.380156+0000","last_clean":"2024-04-24T16:00:25.380156+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:00:25.380156+0000","last_undegraded":"2024-04-24T16:00:25.380156+0000","last_fullsized":"2024-04-24T16:00:25.380156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":36,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.376669+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:25.376669+0000","last_peered":"2024-04-24T16:00:25.376669+0000","last_clean":"2024-04-24T16:00:25.376669+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:25.376669+0000","last_undegraded":"2024-04-24T16:00:25.376669+0000","last_fullsized":"2024-04-24T16:00:25.376669+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":95,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377482+0000","last_change":"2024-04-24T15:59:48.255715+0000","last_active":"2024-04-24T16:00:25.377482+0000","last_peered":"2024-04-24T16:00:25.377482+0000","last_clean":"2024-04-24T16:00:25.377482+0000","last_became_active":"2024-04-24T15:58:43.032284+0000","last_became_peered":"2024-04-24T15:58:43.032284+0000","last_unstale":"2024-04-24T16:00:25.377482+0000","last_undegraded":"2024-04-24T16:00:25.377482+0000","last_fullsized":"2024-04-24T16:00:25.377482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.380030+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:25.380030+0000","last_peered":"2024-04-24T16:00:25.380030+0000","last_clean":"2024-04-24T16:00:25.380030+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:25.380030+0000","last_undegraded":"2024-04-24T16:00:25.380030+0000","last_fullsized":"2024-04-24T16:00:25.380030+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377179+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:25.377179+0000","last_peered":"2024-04-24T16:00:25.377179+0000","last_clean":"2024-04-24T16:00:25.377179+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:25.377179+0000","last_undegraded":"2024-04-24T16:00:25.377179+0000","last_fullsized":"2024-04-24T16:00:25.377179+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":91,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377157+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:25.377157+0000","last_peered":"2024-04-24T16:00:25.377157+0000","last_clean":"2024-04-24T16:00:25.377157+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:25.377157+0000","last_undegraded":"2024-04-24T16:00:25.377157+0000","last_fullsized":"2024-04-24T16:00:25.377157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":213,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.388692+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T16:00:25.388692+0000","last_peered":"2024-04-24T16:00:25.388692+0000","last_clean":"2024-04-24T16:00:25.388692+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T16:00:25.388692+0000","last_undegraded":"2024-04-24T16:00:25.388692+0000","last_fullsized":"2024-04-24T16:00:25.388692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"59'44","reported_seq":287,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.376833+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:25.376833+0000","last_peered":"2024-04-24T16:00:25.376833+0000","last_clean":"2024-04-24T16:00:25.376833+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:25.376833+0000","last_undegraded":"2024-04-24T16:00:25.376833+0000","last_fullsized":"2024-04-24T16:00:25.376833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":282,"num_read_kb":10860,"num_write":55,"num_write_kb":8225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.388718+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:25.388718+0000","last_peered":"2024-04-24T16:00:25.388718+0000","last_clean":"2024-04-24T16:00:25.388718+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:25.388718+0000","last_undegraded":"2024-04-24T16:00:25.388718+0000","last_fullsized":"2024-04-24T16:00:25.388718+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.380054+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:25.380054+0000","last_peered":"2024-04-24T16:00:25.380054+0000","last_clean":"2024-04-24T16:00:25.380054+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:25.380054+0000","last_undegraded":"2024-04-24T16:00:25.380054+0000","last_fullsized":"2024-04-24T16:00:25.380054+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"68'274","reported_seq":1020,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.394403+0000","last_change":"2024-04-24T16:00:25.382961+0000","last_active":"2024-04-24T16:00:25.394403+0000","last_peered":"2024-04-24T16:00:25.394403+0000","last_clean":"2024-04-24T16:00:25.394403+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:00:25.394403+0000","last_undegraded":"2024-04-24T16:00:25.394403+0000","last_fullsized":"2024-04-24T16:00:25.394403+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:38:07.367701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":25165862,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1675,"num_read_kb":46601,"num_write":265,"num_write_kb":41100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":35,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.388688+0000","last_change":"2024-04-24T15:59:52.896635+0000","last_active":"2024-04-24T16:00:25.388688+0000","last_peered":"2024-04-24T16:00:25.388688+0000","last_clean":"2024-04-24T16:00:25.388688+0000","last_became_active":"2024-04-24T15:59:50.311714+0000","last_became_peered":"2024-04-24T15:59:50.311714+0000","last_unstale":"2024-04-24T16:00:25.388688+0000","last_undegraded":"2024-04-24T16:00:25.388688+0000","last_fullsized":"2024-04-24T16:00:25.388688+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:57.999783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037330000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":79,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.388774+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:25.388774+0000","last_peered":"2024-04-24T16:00:25.388774+0000","last_clean":"2024-04-24T16:00:25.388774+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:25.388774+0000","last_undegraded":"2024-04-24T16:00:25.388774+0000","last_fullsized":"2024-04-24T16:00:25.388774+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"68'152","reported_seq":693,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.395989+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:25.395989+0000","last_peered":"2024-04-24T16:00:25.395989+0000","last_clean":"2024-04-24T16:00:25.395989+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:25.395989+0000","last_undegraded":"2024-04-24T16:00:25.395989+0000","last_fullsized":"2024-04-24T16:00:25.395989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:21.334067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":26214495,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":480,"num_read_kb":58606,"num_write":139,"num_write_kb":50052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":54,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377388+0000","last_change":"2024-04-24T16:00:22.522734+0000","last_active":"2024-04-24T16:00:25.377388+0000","last_peered":"2024-04-24T16:00:25.377388+0000","last_clean":"2024-04-24T16:00:25.377388+0000","last_became_active":"2024-04-24T16:00:22.364893+0000","last_became_peered":"2024-04-24T16:00:22.364893+0000","last_unstale":"2024-04-24T16:00:25.377388+0000","last_undegraded":"2024-04-24T16:00:25.377388+0000","last_fullsized":"2024-04-24T16:00:25.377388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.344126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.396383+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:25.396383+0000","last_peered":"2024-04-24T16:00:25.396383+0000","last_clean":"2024-04-24T16:00:25.396383+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:25.396383+0000","last_undegraded":"2024-04-24T16:00:25.396383+0000","last_fullsized":"2024-04-24T16:00:25.396383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'101","reported_seq":877,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.380060+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:25.380060+0000","last_peered":"2024-04-24T16:00:25.380060+0000","last_clean":"2024-04-24T16:00:25.380060+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:25.380060+0000","last_undegraded":"2024-04-24T16:00:25.380060+0000","last_fullsized":"2024-04-24T16:00:25.380060+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:41.233351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":20971589,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":903,"num_read_kb":44881,"num_write":75,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":39,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377002+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:25.377002+0000","last_peered":"2024-04-24T16:00:25.377002+0000","last_clean":"2024-04-24T16:00:25.377002+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:25.377002+0000","last_undegraded":"2024-04-24T16:00:25.377002+0000","last_fullsized":"2024-04-24T16:00:25.377002+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"68'5","reported_seq":81,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.393912+0000","last_change":"2024-04-24T15:59:48.261056+0000","last_active":"2024-04-24T16:00:25.393912+0000","last_peered":"2024-04-24T16:00:25.393912+0000","last_clean":"2024-04-24T16:00:25.393912+0000","last_became_active":"2024-04-24T15:58:43.030599+0000","last_became_peered":"2024-04-24T15:58:43.030599+0000","last_unstale":"2024-04-24T16:00:25.393912+0000","last_undegraded":"2024-04-24T16:00:25.393912+0000","last_fullsized":"2024-04-24T16:00:25.393912+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:18.386081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"59'8","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.396398+0000","last_change":"2024-04-24T16:00:22.470388+0000","last_active":"2024-04-24T16:00:25.396398+0000","last_peered":"2024-04-24T16:00:25.396398+0000","last_clean":"2024-04-24T16:00:25.396398+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:00:25.396398+0000","last_undegraded":"2024-04-24T16:00:25.396398+0000","last_fullsized":"2024-04-24T16:00:25.396398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165877,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":42,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T16:00:25.377200+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:25.377200+0000","last_peered":"2024-04-24T16:00:25.377200+0000","last_clean":"2024-04-24T16:00:25.377200+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:25.377200+0000","last_undegraded":"2024-04-24T16:00:25.377200+0000","last_fullsized":"2024-04-24T16:00:25.377200+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":84934896,"num_objects":41,"num_object_clones":1,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":7,"num_read":3340,"num_read_kb":160948,"num_write":534,"num_write_kb":132148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169934848,"data_stored":176161248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":571,"ondisk_log_size":571,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217933,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":112,"num_read_kb":4807,"num_write":44,"num_write_kb":131211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748827,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268484608,"data_stored":268435866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364724,"kb_used_data":67368,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94007116,"statfs":{"total":96636764160,"available":96263286784,"internally_reserved":0,"allocated":68984832,"data_stored":71628220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.36799999999999999}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35299999999999998}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.31},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.434}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.33800000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371896,"kb_used_data":74540,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999944,"statfs":{"total":96636764160,"available":96255942656,"internally_reserved":0,"allocated":76328960,"data_stored":75827833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.443},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.33100000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.378}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.309},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.27300000000000002}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36399999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640282,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347892,"kb_used_data":50536,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94023948,"statfs":{"total":96636764160,"available":96280522752,"internally_reserved":0,"allocated":51748864,"data_stored":51252346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47499999999999998}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51100000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.61599999999999999}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58299999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640281,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363696,"kb_used_data":62240,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94008144,"statfs":{"total":96636764160,"available":96264339456,"internally_reserved":0,"allocated":63733760,"data_stored":63239593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.42899999999999999}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57299999999999995}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64200000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640282,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388288,"kb_used_data":86828,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93983552,"statfs":{"total":96636764160,"available":96239157248,"internally_reserved":0,"allocated":88911872,"data_stored":88405490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.42599999999999999}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.50700000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58199999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393988,"kb_used_data":92532,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93977852,"statfs":{"total":96636764160,"available":96233320448,"internally_reserved":0,"allocated":94752768,"data_stored":97389763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45900000000000002}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55700000000000005}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.58899999999999997}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.629},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37761024,"data_stored":37748827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51396608,"data_stored":54526085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25169920,"data_stored":25165862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26226688,"data_stored":29360223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:29.578 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T16:00:29.579 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:29.579 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.21338347442614974 2024-04-24T16:00:29.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 0.21338347442614974 2024-04-24T16:00:29.618 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:29.620+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:29.641 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:29.641 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:29.641 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:29.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:29.724 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:29.720+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:29.774 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:29.774 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:29.775 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:29.775 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:29.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:29.828 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:29.830+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:29.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:29.908 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.603911056999998, 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.602764270999998, 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.586912149, 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.564147517999999, 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.554029161999999, 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.526086047, 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.526068221000003, 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.516450823, 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.512721912000003, 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.486837848999997, 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.430616209, 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.430595099000001, 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.317720254000001, 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.308302484000002, 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.291786318, 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.291229901999998, 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.241999497999998, 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.230321670999999, 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.228973912000001, 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.220846332000001, 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:30.001 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:30.002 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:30.004+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:30.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:30.046 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:30.066 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:30.085 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:30.103 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:30.105+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:30.105 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:30.124 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:30.144 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:30.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:30.169 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.169 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:30.204 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:30.206+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:30.303 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.303 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.304 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:30.306+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:30.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 0.213383 (36a0) 2024-04-24T16:00:30.405 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:30.407+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:30.442 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.442 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:30.507 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:30.509+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:30.604 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.604 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.607 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:30.610+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:30.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:30.709 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:30.710+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:30.755 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.755 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:30.810 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:30.812+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:30.910 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:30.912+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:30.949 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:30.949 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:30.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:31.046 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:31.048+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:31.111 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:31.114+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:31.144 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:31.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:31.243 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:31.245+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:31.313 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:31.315+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:31.341 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:31.342 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:31.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:31.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:00:31.437 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:31.439+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:31.516 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:31.517+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:31.544 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:31.544 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:31.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:31.617 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:31.619+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:31.703 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:31.703 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:31.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:31.718 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:31.719+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:31.818 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:31.820+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:31.853 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:31.854 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:31.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:31.919 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:31.920+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:32.019 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:32.021+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:32.047 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:32.200 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:32.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:32.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:32.203 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:32.203+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:32.302 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:32.304+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:32.335 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:32.335 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:32.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:32.407 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:32.411 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:00:32.428 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:32.429 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:32.431+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:32.447 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:32.467 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:32.487 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:32.504 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:32.506+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:32.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:32.512 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:32.512 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:32.512 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:32.512 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:32.512 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:32.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:32.605 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:32.607+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.377506441999998, 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.369540411999999, 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.369506217000001, 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.369487579000001, 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:32.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:32.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:32.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:32.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.362164841999999, 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.340505819000001, 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:32.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:32.714 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.336257185000001, 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:32.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.336222667000001, 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:32.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:32.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.722 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.306061563, 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.254966623999998, 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.201448112000001, 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.078107952000003, 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.061238496999998, 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.053022071999997, 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.039481924, 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.037597255000001, 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.752 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:32.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:32.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:32.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:32.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.016988996000002, 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:32.757 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:32.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:32.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:32.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.007067495999998, 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.762 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:32.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:32.766 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.002419238000002, 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:32.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.000925447, 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:32.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:32.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:32.775 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:32.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:32.777 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:32.778+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:32.877 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:32.878+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:32.941 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:32.941 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:32.941 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:32.941 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:32.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:32.979 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:32.981+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:33.080 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:33.081+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:33.114 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:33.114 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:33.114 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:33.115 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:33.115 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:33.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:33.181 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:33.183+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:33.281 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:33.284+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.985333382, 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:33.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.984186596000001, 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:33.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.968334474000002, 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.945569843000001, 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.935451487000002, 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.907508372000002, 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.907490545999998, 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.317 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:33.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:33.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:33.322 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.897873148000002, 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:33.326 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.894144236999999, 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.868260174, 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.812038534000003, 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.812017423999997, 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.699142578999997, 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.689724808999998, 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.673208643000002, 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.672652227, 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.623421823000001, 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.611743996000001, 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.610396237000003, 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.602268657000003, 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.378 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.379 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:33.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:33.382 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:33.384+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:33.483 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:33.485+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:33.503 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:33.503 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:33.503 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:33.503 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:33.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:33.584 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:33.586+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:33.636 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:33.636 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:33.636 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:33.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:33.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:33.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:33.685 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:33.687+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:33.786 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:33.788+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.488421731999999, 2024-04-24T16:00:33.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:33.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.486617152000001, 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:33.816 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:33.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.486585783999999, 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:33.825 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.462862057999999, 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:33.829 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.435004319999997, 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.430374815, 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:33.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.423567272, 2024-04-24T16:00:33.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:33.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:33.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.842 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.393027044999997, 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:33.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:33.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.846 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.377387364999997, 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.850 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.338212116000001, 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.325593996000002, 2024-04-24T16:00:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.203197858999999, 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.189668650999998, 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.178934323, 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.164638179999997, 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.875 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:33.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:33.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.880 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.132411087999998, 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.885 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.126315247000001, 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:33.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:33.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:33.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.122860690000003, 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:33.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 45.087416568999998, 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:33.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 38.035956804000001, 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:33.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:33.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:33.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.903 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:33.904 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:33.904 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:33.905 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:33.907+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:33.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:34.006 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:34.007+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:34.055 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:34.055 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:34.055 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:34.055 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:34.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:34.111 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:34.113+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:34.205 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:34.206 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:34.208 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:34.210+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:34.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:34.309 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:34.311+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.083227285, 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.082075607, 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.067055822999997, 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.046123485000003, 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.045501530999999, 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.017517009999999, 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 46.00420132, 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.991489518999998, 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.932900613999998, 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.913465158000001, 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.900985175999999, 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.773550127999997, 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.772863381000001, 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:34.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:34.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.771708906999997, 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:34.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:34.463 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.768061482, 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:34.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:34.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.467 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.748101390999999, 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.746693481999998, 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.473 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.746057432999997, 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:34.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:34.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:34.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.562201923000003, 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.486 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.923458335999999, 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.490 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:34.491 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:34.491 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:34.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T16:00:34.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:34.493 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:34.494+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:34.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:34.594 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:34.595+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:34.691 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:34.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:34.691 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:34.691 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:34.694 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:34.696+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:34.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:34.715 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:34.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:34.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:34.796 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:34.797+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:34.815 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:34.853 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:34.854 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:34.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:34.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:34.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:34.895 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:34.908 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:34.909+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:34.998 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:35.000+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:35.032 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:35.032 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:35.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:35.128 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:35.130+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:35.201 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:35.203+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:35.226 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:35.227 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:35.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:35.322 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:35.324+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:35.403 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:35.404+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:35.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:35.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:35.418 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T16:00:35.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T16:00:35.422 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:35.422 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T16:00:35.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:35.503 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:35.505+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:35.604 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:35.606+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:35.621 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:35.621 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:35.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:35.705 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:35.707+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:35.771 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:35.771 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 74, 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c4fc9996-5ff9-4e71-b213-9020967fae1a", 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-04-24T15:58:30.292182+0000", 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-04-24T16:00:35.414668+0000", 2024-04-24T16:00:35.778 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-04-24T15:58:41.011852+0000", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-04-24T15:58:32.987203+0000", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5801984, 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T16:00:35.779 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 4, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 6, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "luminous", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "quincy", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.444445+0000", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T16:00:35.780 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T16:00:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "67", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 6, 2024-04-24T16:00:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 67, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [ 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: 3 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": 3, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": 3, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T16:00:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T16:00:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:41.808270+0000", 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-24T16:00:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 1, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 1, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 1, 2024-04-24T16:00:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 1, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 1, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "14", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T16:00:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T16:00:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T16:00:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 3, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cache", 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:58:48.277449+0000", 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8201, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T16:00:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 4, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 4, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 4, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 4, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 4, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "67", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "18", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "18", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "18", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 6, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 67, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": 1, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T16:00:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "writeback", 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 250, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bloom", 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "target_size": 0, 2024-04-24T16:00:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "seed": 0 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 60, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 8, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [ 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: 1000000 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 4, 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "new_default_pool37497", 2024-04-24T16:00:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-04-24T15:59:49.072101+0000", 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2024-04-24T16:00:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "48", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T16:00:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 3, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 48, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-04-24T16:00:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-04-24T16:00:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "51c39890-e79b-4030-a4a0-8c98ec2018ba", 2024-04-24T16:00:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 71, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-04-24T16:00:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41951 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/41951", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/41951", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6819/41951", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/41951", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "1fb2523f-db36-46d9-80c4-dc08ac530daa", 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 71, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41952 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/41952", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/41952", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/41952", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/41952", 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7d4dd13f-26d8-4c39-a6a6-020e91634f4e", 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-04-24T16:00:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41953 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6814/41953", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6818/41953", 2024-04-24T16:00:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/41953", 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/41953", 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "03ed1eb6-8fa9-497c-b769-2743acc0e6aa", 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0.21337890625, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6801", 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41923 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/41923", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6807/41923", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6815/41923", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6811/41923", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f7ff11ec-62d6-44f3-bea1-7ff62d280e2f", 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 64, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2024-04-24T16:00:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41924 2024-04-24T16:00:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6817/41924", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6819/41924", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6823/41924", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6821/41924", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "423ca5ac-f6a3-496e-a4c5-9ac205e2e474", 2024-04-24T16:00:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 11, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 71, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6800", 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-24T16:00:35.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 41925 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6802/41925", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6806/41925", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6814/41925", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6810/41925", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.832 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.171683+0000", 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.085846+0000", 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.118237+0000", 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.172469+0000", 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.139953+0000", 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540138320759226367, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-04-24T15:58:38.084518+0000", 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.836 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [ 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "4.7", 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: 3, 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: 4 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": {}, 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-04-24T16:00:35.837 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-04-24T16:00:35.838 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:35.839 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:35.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:35.841 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:35.842+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:35.842 INFO:tasks.thrashosds.thrasher:j is {'epoch': 74, 'fsid': 'c4fc9996-5ff9-4e71-b213-9020967fae1a', 'created': '2024-04-24T15:58:30.292182+0000', 'modified': '2024-04-24T16:00:35.414668+0000', 'last_up_change': '2024-04-24T15:58:41.011852+0000', 'last_in_change': '2024-04-24T15:58:32.987203+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-24T15:58:41.444445+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-24T15:58:41.808270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T15:58:48.277449+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '18', 'last_force_op_resend_prenautilus': '18', 'last_force_op_resend_preluminous': '18', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool37497', 'create_time': '2024-04-24T15:59:49.072101+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '51c39890-e79b-4030-a4a0-8c98ec2018ba', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 41951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 41951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6817', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 41951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6813', 'nonce': 41951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 41951}]}, 'public_addr': '172.21.15.32:6809/41951', 'cluster_addr': '172.21.15.32:6811/41951', 'heartbeat_back_addr': '172.21.15.32:6819/41951', 'heartbeat_front_addr': '172.21.15.32:6815/41951', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fb2523f-db36-46d9-80c4-dc08ac530daa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 41952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 41952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 41952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 41952}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 41952}]}, 'public_addr': '172.21.15.32:6801/41952', 'cluster_addr': '172.21.15.32:6803/41952', 'heartbeat_back_addr': '172.21.15.32:6807/41952', 'heartbeat_front_addr': '172.21.15.32:6805/41952', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7d4dd13f-26d8-4c39-a6a6-020e91634f4e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6814', 'nonce': 41953}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6818', 'nonce': 41953}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 41953}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 41953}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 41953}]}, 'public_addr': '172.21.15.32:6814/41953', 'cluster_addr': '172.21.15.32:6818/41953', 'heartbeat_back_addr': '172.21.15.32:6823/41953', 'heartbeat_front_addr': '172.21.15.32:6821/41953', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '03ed1eb6-8fa9-497c-b769-2743acc0e6aa', 'up': 1, 'in': 1, 'weight': 0.21337890625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6803', 'nonce': 41923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6807', 'nonce': 41923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6815', 'nonce': 41923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 41923}, {'type': 'v1', 'addr': '172.21.15.86:6811', 'nonce': 41923}]}, 'public_addr': '172.21.15.86:6803/41923', 'cluster_addr': '172.21.15.86:6807/41923', 'heartbeat_back_addr': '172.21.15.86:6815/41923', 'heartbeat_front_addr': '172.21.15.86:6811/41923', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7ff11ec-62d6-44f3-bea1-7ff62d280e2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6816', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6817', 'nonce': 41924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6818', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6819', 'nonce': 41924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6822', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6823', 'nonce': 41924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6820', 'nonce': 41924}, {'type': 'v1', 'addr': '172.21.15.86:6821', 'nonce': 41924}]}, 'public_addr': '172.21.15.86:6817/41924', 'cluster_addr': '172.21.15.86:6819/41924', 'heartbeat_back_addr': '172.21.15.86:6823/41924', 'heartbeat_front_addr': '172.21.15.86:6821/41924', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '423ca5ac-f6a3-496e-a4c5-9ac205e2e474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6802', 'nonce': 41925}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6806', 'nonce': 41925}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6814', 'nonce': 41925}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 41925}, {'type': 'v1', 'addr': '172.21.15.86:6810', 'nonce': 41925}]}, 'public_addr': '172.21.15.86:6802/41925', 'cluster_addr': '172.21.15.86:6806/41925', 'heartbeat_back_addr': '172.21.15.86:6814/41925', 'heartbeat_front_addr': '172.21.15.86:6810/41925', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.171683+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.085846+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.118237+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.172469+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.139953+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T15:58:38.084518+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.7', 'osds': [3, 4]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T16:00:35.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:35.941 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:35.943+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:36.005 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.005 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:36.056 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:36.058+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:36.144 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:36.146+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:36.179 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.179 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:36.238 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:36.239 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:36.244 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:36.246+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:36.251 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-04-24T16:00:35.771824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220791843,"num_objects":92,"num_object_clones":2,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":7,"num_read":3761,"num_read_kb":166010,"num_write":755,"num_write_kb":267831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":96469232,"num_keys_recovered":26,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2199956,"kb_used_data":403508,"kb_used_omap":9,"kb_used_meta":1796342,"kb_avail":564031084,"statfs":{"total":579820584960,"available":577567830016,"internally_reserved":0,"allocated":413192192,"data_stored":422663347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1839454224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":155,"num_read_kb":125,"num_write":12,"num_write_kb":2050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":33554508,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.372695"},"pg_stats":[{"pgid":"1.5","version":"44'4","reported_seq":111,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.404926+0000","last_change":"2024-04-24T16:00:22.519701+0000","last_active":"2024-04-24T16:00:30.404926+0000","last_peered":"2024-04-24T16:00:30.404926+0000","last_clean":"2024-04-24T16:00:30.404926+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:00:30.404926+0000","last_undegraded":"2024-04-24T16:00:30.404926+0000","last_fullsized":"2024-04-24T16:00:30.404926+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400445+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:30.400445+0000","last_peered":"2024-04-24T16:00:30.400445+0000","last_clean":"2024-04-24T16:00:30.400445+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:30.400445+0000","last_undegraded":"2024-04-24T16:00:30.400445+0000","last_fullsized":"2024-04-24T16:00:30.400445+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":127,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-24T16:00:31.523686+0000","last_change":"2024-04-24T16:00:31.523686+0000","last_active":"2024-04-24T16:00:31.523686+0000","last_peered":"2024-04-24T16:00:31.523686+0000","last_clean":"2024-04-24T16:00:31.523686+0000","last_became_active":"2024-04-24T16:00:31.414295+0000","last_became_peered":"2024-04-24T16:00:31.414295+0000","last_unstale":"2024-04-24T16:00:31.523686+0000","last_undegraded":"2024-04-24T16:00:31.523686+0000","last_fullsized":"2024-04-24T16:00:31.523686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:31.661486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":11,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.404709+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:30.404709+0000","last_peered":"2024-04-24T16:00:30.404709+0000","last_clean":"2024-04-24T16:00:30.404709+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:30.404709+0000","last_undegraded":"2024-04-24T16:00:30.404709+0000","last_fullsized":"2024-04-24T16:00:30.404709+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400441+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:30.400441+0000","last_peered":"2024-04-24T16:00:30.400441+0000","last_clean":"2024-04-24T16:00:30.400441+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:30.400441+0000","last_undegraded":"2024-04-24T16:00:30.400441+0000","last_fullsized":"2024-04-24T16:00:30.400441+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"44'5","reported_seq":94,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400376+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:30.400376+0000","last_peered":"2024-04-24T16:00:30.400376+0000","last_clean":"2024-04-24T16:00:30.400376+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:30.400376+0000","last_undegraded":"2024-04-24T16:00:30.400376+0000","last_fullsized":"2024-04-24T16:00:30.400376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":216,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.402994+0000","last_change":"2024-04-24T15:58:43.031714+0000","last_active":"2024-04-24T16:00:30.402994+0000","last_peered":"2024-04-24T16:00:30.402994+0000","last_clean":"2024-04-24T16:00:30.402994+0000","last_became_active":"2024-04-24T15:58:43.031106+0000","last_became_peered":"2024-04-24T15:58:43.031106+0000","last_unstale":"2024-04-24T16:00:30.402994+0000","last_undegraded":"2024-04-24T16:00:30.402994+0000","last_fullsized":"2024-04-24T16:00:30.402994+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:25:54.102741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"59'44","reported_seq":290,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400432+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:30.400432+0000","last_peered":"2024-04-24T16:00:30.400432+0000","last_clean":"2024-04-24T16:00:30.400432+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:30.400432+0000","last_undegraded":"2024-04-24T16:00:30.400432+0000","last_fullsized":"2024-04-24T16:00:30.400432+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:01.234263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":282,"num_read_kb":10860,"num_write":55,"num_write_kb":8225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.403364+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:30.403364+0000","last_peered":"2024-04-24T16:00:30.403364+0000","last_clean":"2024-04-24T16:00:30.403364+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:30.403364+0000","last_undegraded":"2024-04-24T16:00:30.403364+0000","last_fullsized":"2024-04-24T16:00:30.403364+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":127,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.404774+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:30.404774+0000","last_peered":"2024-04-24T16:00:30.404774+0000","last_clean":"2024-04-24T16:00:30.404774+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:30.404774+0000","last_undegraded":"2024-04-24T16:00:30.404774+0000","last_fullsized":"2024-04-24T16:00:30.404774+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"69'282","reported_seq":1091,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400818+0000","last_change":"2024-04-24T16:00:25.382961+0000","last_active":"2024-04-24T16:00:30.400818+0000","last_peered":"2024-04-24T16:00:30.400818+0000","last_clean":"2024-04-24T16:00:30.400818+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:00:30.400818+0000","last_undegraded":"2024-04-24T16:00:30.400818+0000","last_fullsized":"2024-04-24T16:00:30.400818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:38:07.367701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":25165862,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1821,"num_read_kb":46720,"num_write":267,"num_write_kb":41102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-24T16:00:31.657960+0000","last_change":"2024-04-24T16:00:31.657960+0000","last_active":"2024-04-24T16:00:31.657960+0000","last_peered":"2024-04-24T16:00:31.657960+0000","last_clean":"2024-04-24T16:00:31.657960+0000","last_became_active":"2024-04-24T16:00:31.657630+0000","last_became_peered":"2024-04-24T16:00:31.657630+0000","last_unstale":"2024-04-24T16:00:31.657960+0000","last_undegraded":"2024-04-24T16:00:31.657960+0000","last_fullsized":"2024-04-24T16:00:31.657960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:57.306064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"44'5","reported_seq":82,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.403131+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:30.403131+0000","last_peered":"2024-04-24T16:00:30.403131+0000","last_clean":"2024-04-24T16:00:30.403131+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:30.403131+0000","last_undegraded":"2024-04-24T16:00:30.403131+0000","last_fullsized":"2024-04-24T16:00:30.403131+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777254,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"69'163","reported_seq":710,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400631+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:30.400631+0000","last_peered":"2024-04-24T16:00:30.400631+0000","last_clean":"2024-04-24T16:00:30.400631+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:30.400631+0000","last_undegraded":"2024-04-24T16:00:30.400631+0000","last_fullsized":"2024-04-24T16:00:30.400631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:21.334067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":27263071,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":484,"num_read_kb":58610,"num_write":149,"num_write_kb":52100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400376+0000","last_change":"2024-04-24T16:00:22.522734+0000","last_active":"2024-04-24T16:00:30.400376+0000","last_peered":"2024-04-24T16:00:30.400376+0000","last_clean":"2024-04-24T16:00:30.400376+0000","last_became_active":"2024-04-24T16:00:22.364893+0000","last_became_peered":"2024-04-24T16:00:22.364893+0000","last_unstale":"2024-04-24T16:00:30.400376+0000","last_undegraded":"2024-04-24T16:00:30.400376+0000","last_fullsized":"2024-04-24T16:00:30.400376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.344126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.403104+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:30.403104+0000","last_peered":"2024-04-24T16:00:30.403104+0000","last_clean":"2024-04-24T16:00:30.403104+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:30.403104+0000","last_undegraded":"2024-04-24T16:00:30.403104+0000","last_fullsized":"2024-04-24T16:00:30.403104+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'101","reported_seq":884,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.404850+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:30.404850+0000","last_peered":"2024-04-24T16:00:30.404850+0000","last_clean":"2024-04-24T16:00:30.404850+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:30.404850+0000","last_undegraded":"2024-04-24T16:00:30.404850+0000","last_fullsized":"2024-04-24T16:00:30.404850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:41.233351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":20971589,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":909,"num_read_kb":44883,"num_write":75,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400644+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:30.400644+0000","last_peered":"2024-04-24T16:00:30.400644+0000","last_clean":"2024-04-24T16:00:30.400644+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:30.400644+0000","last_undegraded":"2024-04-24T16:00:30.400644+0000","last_fullsized":"2024-04-24T16:00:30.400644+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"68'5","reported_seq":100,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-24T16:00:31.701555+0000","last_change":"2024-04-24T16:00:31.701555+0000","last_active":"2024-04-24T16:00:31.701555+0000","last_peered":"2024-04-24T16:00:31.701555+0000","last_clean":"2024-04-24T16:00:31.701555+0000","last_became_active":"2024-04-24T16:00:31.657902+0000","last_became_peered":"2024-04-24T16:00:31.657902+0000","last_unstale":"2024-04-24T16:00:31.701555+0000","last_undegraded":"2024-04-24T16:00:31.701555+0000","last_fullsized":"2024-04-24T16:00:31.701555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:57.117033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388684,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"59'8","reported_seq":128,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.404210+0000","last_change":"2024-04-24T16:00:22.470388+0000","last_active":"2024-04-24T16:00:30.404210+0000","last_peered":"2024-04-24T16:00:30.404210+0000","last_clean":"2024-04-24T16:00:30.404210+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:00:30.404210+0000","last_undegraded":"2024-04-24T16:00:30.404210+0000","last_fullsized":"2024-04-24T16:00:30.404210+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165877,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-24T16:00:30.400527+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:30.400527+0000","last_peered":"2024-04-24T16:00:30.400527+0000","last_clean":"2024-04-24T16:00:30.400527+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:30.400527+0000","last_undegraded":"2024-04-24T16:00:30.400527+0000","last_fullsized":"2024-04-24T16:00:30.400527+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":85983472,"num_objects":42,"num_object_clones":2,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":3496,"num_read_kb":161073,"num_write":546,"num_write_kb":134198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032000,"data_stored":184549856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134217933,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":111,"num_read_kb":4807,"num_write":44,"num_write_kb":131211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":71303335,"num_keys_recovered":12,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268484608,"data_stored":268435866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365788,"kb_used_data":68428,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94006052,"statfs":{"total":96636764160,"available":96262197248,"internally_reserved":0,"allocated":70070272,"data_stored":75838636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.55500000000000005}]},{"osd":2,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.498},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Apr 24 15:59:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36699999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640282,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371924,"kb_used_data":74564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93999916,"statfs":{"total":96636764160,"available":96255913984,"internally_reserved":0,"allocated":76353536,"data_stored":75838598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.376}]},{"osd":1,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.433},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48999999999999999}]},{"osd":3,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.314}]},{"osd":5,"last update":"Wed Apr 24 15:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.39900000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640283,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315160,"kb_used_data":17796,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94056680,"statfs":{"total":96636764160,"available":96314040320,"internally_reserved":0,"allocated":18223104,"data_stored":17713974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.375}]},{"osd":1,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.35099999999999998}]},{"osd":4,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.33400000000000002}]},{"osd":5,"last update":"Wed Apr 24 15:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.248},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28199999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363728,"kb_used_data":62276,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94008112,"statfs":{"total":96636764160,"available":96264306688,"internally_reserved":0,"allocated":63770624,"data_stored":63255705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.27000000000000002}]},{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32900000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.436}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42699999999999999}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41399999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640283,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388320,"kb_used_data":86864,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93983520,"statfs":{"total":96636764160,"available":96239124480,"internally_reserved":0,"allocated":88948736,"data_stored":88421602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.34200000000000003}]},{"osd":2,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.38100000000000001}]},{"osd":3,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.40799999999999997}]},{"osd":4,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.443}]},{"osd":5,"last update":"Wed Apr 24 15:59:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48099999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395036,"kb_used_data":93580,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93976804,"statfs":{"total":96636764160,"available":96232247296,"internally_reserved":0,"allocated":95825920,"data_stored":101594832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.31},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40699999999999997}]},{"osd":3,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.498},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 15:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37761024,"data_stored":37748827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16781312,"data_stored":16777254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67117056,"data_stored":67108917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52445184,"data_stored":58720389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25169920,"data_stored":25165862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33570816,"data_stored":33554539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27275264,"data_stored":33554527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:36.253 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [1, 2, 0, 3] 2024-04-24T16:00:36.254 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '1', '2', '0', '3'] 2024-04-24T16:00:36.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.0 1 2 0 3 2024-04-24T16:00:36.313 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.314 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:36.345 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:36.347+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:36.436 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-04-24T16:00:36.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:36.442 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:36.442 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:36.442 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:36.443 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:36.445 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:36.445 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:36.445 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T16:00:36.446 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:36.447+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:36.469 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:36.469 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:36.469 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011241 s, 93.3 MB/s 2024-04-24T16:00:36.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 39874 2024-04-24T16:00:36.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:36.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=39874 2024-04-24T16:00:36.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:36.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:36.472 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:36.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:36.473 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:36.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:36.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:36.475 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:36.492 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.492 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:36.586 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:36.588+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:36.647 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:36.649+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:36.665 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.666 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:36.748 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:36.750+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:36.838 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.839 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:36.849 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:36.851+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:36.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:36.953 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:36.955+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:36.971 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:36.971 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:36.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:37.050 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:37.052+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:37.072 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:37.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:37.106 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:37.106 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:37.110 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:37.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:37.131 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:37.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:37.152 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:37.153+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:37.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:37.253 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:37.254+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:37.256 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:37.256 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:37.257 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:37.257 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:37.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:37.371 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:37.363+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:set 2.0 pg_upmap_items mapping to [1->2,0->3] 2024-04-24T16:00:37.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:00:37.462 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:37.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:37.462 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:37.462 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:37.462 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:37.464 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:37.464+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:37.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:37.564 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:37.566+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:37.656 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:37.658+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:37.661 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:37.661 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.344214987999997, 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:37.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:37.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.343063309999998, 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.328043526000002, 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:37.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.307111188, 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.306489233999997, 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.278504712999997, 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.265189022999998, 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.252477222000003, 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.193888317000003, 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:37.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:37.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.174452860999999, 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:37.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:37.703 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.161972878999997, 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.034537831000002, 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:37.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:37.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.033851083999998, 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:37.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:37.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.032696610000002, 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:37.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:37.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:37.721 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.029049184999998, 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:37.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.009089093999997, 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.007681185000003, 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.007045136000002, 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:37.736 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 44.823189626000001, 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.745 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.184446039000001, 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:37.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:37.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:37.749 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:37.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:37.757 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:37.759+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:37.858 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:37.860+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:37.929 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:37.930 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:37.930 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:37.930 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:37.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:37.958 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:37.961+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:38.059 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:38.061+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:38.079 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:38.079 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:38.079 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:38.079 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:38.079 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:38.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:38.160 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:38.163+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:38.262 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:38.263+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:38.275 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:38.275 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:38.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.971015508999997, 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.961395445000001, 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.940994877999998, 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.928063076000001, 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:38.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.928029324999997, 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.293 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.926506914999997, 2024-04-24T16:00:38.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.921095799, 2024-04-24T16:00:38.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:38.305 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.913347281, 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.911217258000001, 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:38.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:38.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:38.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:38.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.895514300999999, 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.314 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:38.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:38.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:38.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.318 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.895486314000003, 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.878715237999998, 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:38.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:38.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:38.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.873371714000001, 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.328 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.861161469000002, 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.853726303999998, 2024-04-24T16:00:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:38.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:38.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.343 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.669470404999998, 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.665234478000002, 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:38.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:38.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.658394712000003, 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.352 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:38.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:38.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:38.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.904338294, 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:38.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:00:38.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:00:38.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:00:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.361 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.902113630000001, 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:38.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:00:38.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.367 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:38.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:38.369 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:38.370+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:38.436 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:00:38.470 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:38.472+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:38.523 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:38.524 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:38.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:38.571 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:38.573+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:38.671 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:38.673+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:38.673 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:38.673 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:38.673 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:38.673 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:38.673 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:38.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:38.772 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:38.774+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.447857031, 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:00:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:00:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.446993684000001, 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:00:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.443611419, 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:00:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.439790737999999, 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014029229999999999, 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387204+0000", 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387733+0000", 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052950899999999995 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387753+0000", 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9383000000000001e-05 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387762+0000", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6409999999999997e-06 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387826+0000", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3915e-05 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387833+0000", 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-06 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.388608+0000", 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077458099999999999 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.436451733, 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480992, 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390543+0000", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391057+0000", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00051421400000000003 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391073+0000", 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519000000000002e-05 2024-04-24T16:00:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391081+0000", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1980000000000001e-06 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391126+0000", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5429000000000002e-05 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391132+0000", 2024-04-24T16:00:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3770000000000004e-06 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.392025+0000", 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089311900000000005 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head, version: 56'185, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head@56'185, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.422407638999999, 2024-04-24T16:00:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001542362, 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404587+0000", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405173+0000", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000586328 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405189+0000", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5630000000000001e-05 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405196+0000", 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2930000000000003e-06 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405249+0000", 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2280000000000001e-05 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405256+0000", 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7200000000000006e-06 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.406130+0000", 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087399199999999998 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.418265759000001, 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013967070000000001, 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408729+0000", 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409290+0000", 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056137799999999996 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409314+0000", 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3377999999999999e-05 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409323+0000", 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3130000000000004e-06 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409384+0000", 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0752999999999999e-05 2024-04-24T16:00:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409389+0000", 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5400000000000003e-06 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.410127+0000", 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073712699999999996 2024-04-24T16:00:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.414897063, 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001435471, 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412098+0000", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412702+0000", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060413800000000005 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412717+0000", 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5545000000000001e-05 2024-04-24T16:00:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412725+0000", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7060000000000004e-06 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412782+0000", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6666000000000003e-05 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412787+0000", 2024-04-24T16:00:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5770000000000001e-06 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.413534+0000", 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074670499999999996 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.40784975, 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015370780000000001, 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419145+0000", 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419734+0000", 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058931100000000002 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419750+0000", 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5996000000000001e-05 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419758+0000", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9629999999999998e-06 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419807+0000", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8251999999999998e-05 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419813+0000", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6590000000000001e-06 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.420683+0000", 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086977599999999995 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.396987934, 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001420075, 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430007+0000", 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430585+0000", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057783400000000005 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430603+0000", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8485e-05 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430612+0000", 2024-04-24T16:00:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6640000000000004e-06 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430679+0000", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7578999999999998e-05 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430686+0000", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0190000000000003e-06 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.431428+0000", 2024-04-24T16:00:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074226900000000005 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.393416081, 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.388382652000001, 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:00:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.384427904000001, 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.380321585000001, 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001428697, 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446673+0000", 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447203+0000", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053015899999999999 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447226+0000", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2578e-05 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447235+0000", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8300000000000002e-06 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447305+0000", 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0418000000000002e-05 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447325+0000", 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9647999999999999e-05 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.448103+0000", 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077794699999999997 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.376508364999999, 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001591912, 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450486+0000", 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451016+0000", 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052945499999999996 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451033+0000", 2024-04-24T16:00:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7416000000000001e-05 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451042+0000", 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1759999999999999e-06 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451098+0000", 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.609e-05 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451106+0000", 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9230000000000002e-06 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.452079+0000", 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097278800000000004 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.370608725, 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016296419999999999, 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456386+0000", 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457118+0000", 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073264499999999998 2024-04-24T16:00:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457144+0000", 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5667999999999999e-05 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457158+0000", 2024-04-24T16:00:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4542000000000001e-05 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457220+0000", 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1935999999999998e-05 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457227+0000", 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2110000000000003e-06 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.458017+0000", 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078992899999999998 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.366694612, 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001562031, 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460300+0000", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460929+0000", 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062916800000000004 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460947+0000", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7985000000000001e-05 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460958+0000", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1345000000000001e-05 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461024+0000", 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5708000000000002e-05 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461032+0000", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.661e-06 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461863+0000", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083151199999999996 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.363042301, 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001413178, 2024-04-24T16:00:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463952+0000", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464492+0000", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053963000000000004 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464509+0000", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7472000000000001e-05 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464521+0000", 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1747e-05 2024-04-24T16:00:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464582+0000", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0881999999999997e-05 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464590+0000", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2269999999999993e-06 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.465367+0000", 2024-04-24T16:00:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077665299999999998 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.359497737, 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:00:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:00:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.444372844, 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:00:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:00:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:38.913 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:38.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:38.915 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:38.917+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:39.015 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:39.016+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:39.052 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:39.052 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:39.052 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:39.052 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:39.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:39.147 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:39.148+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:39.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:39.218 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:39.219+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:39.226 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:39.226 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:39.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:39.319 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:39.321+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:39.349 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:39.369 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:39.388 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.092519074000002, 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.407 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.084553044000003, 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.084518848999998, 2024-04-24T16:00:39.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:39.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:39.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.084500210999998, 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.415 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:39.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:39.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:39.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:39.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.077177474000003, 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.420 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:39.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:39.424 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.055518450999998, 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.051269816999998, 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:39.429 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:39.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.051235298999998, 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.021074194999997, 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.969979256000002, 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.916460743999998, 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:39.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:39.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.793120584, 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:39.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:39.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.776251129000002, 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:39.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:39.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.768034704000002, 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.459 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.754494555999997, 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.752609886999998, 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.732001627999999, 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.722080128000002, 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:39.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:39.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.717431869999999, 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:39.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.484 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 50.715938078999997, 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:39.488 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:39.489 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:39.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:39.492 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:39.492+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:39.590 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:39.593+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:39.617 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:39.617 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:39.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:39.692 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:39.694+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:39.750 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:39.750 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:39.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:39.800 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:39.802+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:39.894 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:39.896+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:39.922 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:39.923 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:39.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:39.995 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:39.997+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:40.096 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:40.098+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:40.100 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.100 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:40.197 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:40.199+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:40.233 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.233 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:40.298 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:40.300+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:40.367 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.367 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:40.399 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:40.400+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:40.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:00:40.499 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:40.501+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:40.532 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.532 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:40.600 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:40.602+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:40.702 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:40.703+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:40.725 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.725 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:40.802 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:40.804+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:40.903 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:40.905+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:40.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:41.004 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:41.006+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:41.068 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:41.069 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:41.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:41.124 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:41.126+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:41.206 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:41.208+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:41.218 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:41.218 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:41.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:41.307 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:41.309+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:41.408 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:41.410+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:41.413 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:41.414 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:41.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:00:41.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:41.509 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:41.511+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:41.589 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:41.590 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:41.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:41.645 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:41.647+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:41.661 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:41.710 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:41.713+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:41.743 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:41.743 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:41.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:41.811 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:41.814+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:41.901 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:41.912 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:41.914+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:41.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:41.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:41.968 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:41.968 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:41.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.589868913, 2024-04-24T16:00:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:00:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:41.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:41.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:00:41.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:00:41.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:00:41.978 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.589005566000001, 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:41.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:41.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:00:41.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:00:41.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:41.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:00:41.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:41.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:00:41.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:00:41.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:41.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:00:41.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:00:41.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:41.988 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.585623300999998, 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:00:41.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:00:41.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:41.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:00:41.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:00:41.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:41.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:41.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.995 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:41.995 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:41.995 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.581802620000001, 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014029229999999999, 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:41.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387205+0000", 2024-04-24T16:00:41.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:41.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:41.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387204+0000", 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387733+0000", 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052950899999999995 2024-04-24T16:00:41.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387753+0000", 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9383000000000001e-05 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387762+0000", 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6409999999999997e-06 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387826+0000", 2024-04-24T16:00:42.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3915e-05 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.387833+0000", 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-06 2024-04-24T16:00:42.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.388608+0000", 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077458099999999999 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.578463615, 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480992, 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.004 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390544+0000", 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.390543+0000", 2024-04-24T16:00:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391057+0000", 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00051421400000000003 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391073+0000", 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519000000000002e-05 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391081+0000", 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1980000000000001e-06 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391126+0000", 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5429000000000002e-05 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.391132+0000", 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3770000000000004e-06 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.392025+0000", 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089311900000000005 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head, version: 56'185, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head@56'185, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.564419521000001, 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001542362, 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404588+0000", 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.404587+0000", 2024-04-24T16:00:42.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405173+0000", 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000586328 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405189+0000", 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5630000000000001e-05 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405196+0000", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2930000000000003e-06 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405249+0000", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2280000000000001e-05 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.405256+0000", 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7200000000000006e-06 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.406130+0000", 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087399199999999998 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.560277640999999, 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013967070000000001, 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408730+0000", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.408729+0000", 2024-04-24T16:00:42.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409290+0000", 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056137799999999996 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409314+0000", 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3377999999999999e-05 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409323+0000", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3130000000000004e-06 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409384+0000", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0752999999999999e-05 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.409389+0000", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5400000000000003e-06 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.410127+0000", 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073712699999999996 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.556908945, 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001435471, 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412099+0000", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412098+0000", 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412702+0000", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060413800000000005 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412717+0000", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5545000000000001e-05 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412725+0000", 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7060000000000004e-06 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412782+0000", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6666000000000003e-05 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.412787+0000", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5770000000000001e-06 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.413534+0000", 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074670499999999996 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head, version: 57'200, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head@57'200, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.549861631999999, 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015370780000000001, 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419146+0000", 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419145+0000", 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419734+0000", 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058931100000000002 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419750+0000", 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5996000000000001e-05 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419758+0000", 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9629999999999998e-06 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419807+0000", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8251999999999998e-05 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.419813+0000", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6590000000000001e-06 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.420683+0000", 2024-04-24T16:00:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086977599999999995 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.538999816, 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001420075, 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430008+0000", 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430007+0000", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430585+0000", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057783400000000005 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430603+0000", 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8485e-05 2024-04-24T16:00:42.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430612+0000", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6640000000000004e-06 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430679+0000", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7578999999999998e-05 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.430686+0000", 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0190000000000003e-06 2024-04-24T16:00:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.431428+0000", 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074226900000000005 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.535427963, 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:00:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:00:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.530394533999999, 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:00:42.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.526439786000001, 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:00:42.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:00:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.522333466999999, 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001428697, 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446674+0000", 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.446673+0000", 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447203+0000", 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053015899999999999 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447226+0000", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2578e-05 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447235+0000", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8300000000000002e-06 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447305+0000", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0418000000000002e-05 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.447325+0000", 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9647999999999999e-05 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.448103+0000", 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077794699999999997 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.518520247000001, 2024-04-24T16:00:42.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001591912, 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450487+0000", 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.450486+0000", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451016+0000", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052945499999999996 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451033+0000", 2024-04-24T16:00:42.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7416000000000001e-05 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451042+0000", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1759999999999999e-06 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451098+0000", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.609e-05 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.451106+0000", 2024-04-24T16:00:42.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9230000000000002e-06 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.452079+0000", 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097278800000000004 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.512620606999999, 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016296419999999999, 2024-04-24T16:00:42.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456387+0000", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.456386+0000", 2024-04-24T16:00:42.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457118+0000", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073264499999999998 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457144+0000", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5667999999999999e-05 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457158+0000", 2024-04-24T16:00:42.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4542000000000001e-05 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457220+0000", 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1935999999999998e-05 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.457227+0000", 2024-04-24T16:00:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2110000000000003e-06 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.458017+0000", 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078992899999999998 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.508706493999998, 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001562031, 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460301+0000", 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460300+0000", 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460929+0000", 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062916800000000004 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460947+0000", 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7985000000000001e-05 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.460958+0000", 2024-04-24T16:00:42.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1345000000000001e-05 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461024+0000", 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5708000000000002e-05 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461032+0000", 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.661e-06 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.461863+0000", 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083151199999999996 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.505054182999999, 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001413178, 2024-04-24T16:00:42.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:42.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463953+0000", 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.463952+0000", 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464492+0000", 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053963000000000004 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464509+0000", 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7472000000000001e-05 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464521+0000", 2024-04-24T16:00:42.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1747e-05 2024-04-24T16:00:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464582+0000", 2024-04-24T16:00:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0881999999999997e-05 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.464590+0000", 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2269999999999993e-06 2024-04-24T16:00:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.465367+0000", 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077665299999999998 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:42.077 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.501509619, 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:42.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.586384725999999, 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:42.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:42.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:42.087 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:42.088 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:42.091+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:42.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:42.089 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:42.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:42.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:42.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:42.189 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:42.192+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:42.226 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:42.227 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:42.227 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:42.227 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:42.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:42.290 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:42.293+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:42.360 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:42.360 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:42.361 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:42.361 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:42.361 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:42.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:42.392 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:42.394+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:42.447 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:42.447 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:42.447 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T16:00:42.447 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:00:42.447 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:00:42.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T16:00:42.497 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:42.500+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:42.544 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.233959876999997, 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:42.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.225993846999998, 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:42.551 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:42.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:42.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:42.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.559 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.225959652, 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:42.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:42.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:42.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:42.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:42.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.569 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.570 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.570 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:42.570 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.225941014, 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.218618276999997, 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.196959253999999, 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.19271062, 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.596 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:42.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:42.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.192676102, 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:42.602 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.162514997999999, 2024-04-24T16:00:42.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:42.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:42.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.111420058999997, 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.615 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.057901547, 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.934561387000002, 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.620 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:42.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.917691931999997, 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:42.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.909475507000003, 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:42.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:42.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:42.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.634 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.895935358999999, 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:42.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.638 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.89405069, 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:42.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.873442431000001, 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.645 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:42.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:42.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.863520930999996, 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.650 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.858872673, 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:42.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:42.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.857378881999999, 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:42.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:42.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:42.662 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:42.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:42.664 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T16:00:42.664 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:42.665 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:42.665+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:42.764 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:42.767+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:42.828 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:42.829 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:42.829 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:42.829 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:42.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:42.879 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:42.882+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:42.966 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:42.969+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:43.002 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:43.002 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:43.002 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:43.002 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:43.002 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:43.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:43.098 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:43.101+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:43.168 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:43.170+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.873232078999997, 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:43.178 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:43.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:43.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.872085292999998, 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.184 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:43.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:43.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.856233171, 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.189 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:43.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:43.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.833468539999998, 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.193 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:43.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:43.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.823350183999999, 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:43.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.795407068999999, 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.202 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:43.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.206 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.795389243000002, 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.785771844999999, 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:43.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.782042934000003, 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:43.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.756158870999997, 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.222 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:43.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:43.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:43.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.699937231, 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:43.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:43.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.699916121000001, 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:43.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.587041276000001, 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:43.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:43.242 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.577623506000002, 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.246 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.56110734, 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.560550923999998, 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:43.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:43.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.257 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.511320519999998, 2024-04-24T16:00:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.259 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.264 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.499642692999998, 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:43.265 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:43.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.498294934, 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.270 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:43.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.490167354, 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.280 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:43.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:43.283 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:43.284+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:43.384 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:43.386+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:43.446 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:43.447 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:43.447 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:43.447 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:43.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:43.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:00:43.509 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:43.511+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:43.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:43.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^39874\b' 2024-04-24T16:00:43.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^39874\b' 2024-04-24T16:00:43.584 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:43.587+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:43.606 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:43.607 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:43.607 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *39874\b' 2024-04-24T16:00:43.607 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *39874\b' 2024-04-24T16:00:43.613 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:43.613 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:43.620 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:43.620 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:43.621 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:43.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:43.685 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:43.687+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:43.787 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:43.789+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:43.796 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.474150141000003, 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:43.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:43.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.472345560999997, 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:43.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:43.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.472314193000003, 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:43.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:43.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:43.811 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.448590467000002, 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:43.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.420732729000001, 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.416103223999997, 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.821 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:43.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.826 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.409295681000003, 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:43.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:43.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.378755454, 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.836 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:43.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:43.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:43.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:43.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:43.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.363115774000001, 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.846 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:43.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:43.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.854 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.854 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:43.854 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.323940524999998, 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:43.860 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.311322404999999, 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.188926268000003, 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:43.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.175397060000002, 2024-04-24T16:00:43.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:43.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:43.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:43.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.164662731999996, 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:43.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:43.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:43.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:43.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:43.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.150366589000001, 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:43.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.118139497000001, 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:43.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:43.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.112043655999997, 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.108589099, 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.906 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.073144978000002, 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:43.910 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 48.021685212999998, 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:43.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:43.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:43.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:43.914 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:43.916 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:43.918+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:43.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:44.017 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:44.019+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:44.056 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:44.057 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:44.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:44.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:44.114 INFO:tasks.workunit.client.0.smithi122.stderr:+ LOG_FILE=/tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:44.115 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n '' ']' 2024-04-24T16:00:44.115 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd37497 --log-file=/tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:44.115 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:44.115 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:44.116 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.116 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:44.117 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:44.120+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:44.118 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:44.119 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:44.119 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd37497 --log-file=/tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:44.218 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:44.220+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:44.219 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:44.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:44.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40075 2024-04-24T16:00:44.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40075 2024-04-24T16:00:44.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40075 -C rbd-nbd 2024-04-24T16:00:44.282 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:44.282 INFO:tasks.workunit.client.0.smithi122.stdout: 40075 ? 00:00:00 rbd-nbd 2024-04-24T16:00:44.283 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mkfs /dev/nbd0 2024-04-24T16:00:44.283 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:44.283 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:44.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.284 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mkfs 2024-04-24T16:00:44.286 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/mkfs 2024-04-24T16:00:44.286 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:44.286 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-04-24T16:00:44.319 INFO:tasks.workunit.client.0.smithi122.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-24T16:00:44.319 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:44.321+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:44.372 INFO:tasks.workunit.client.0.smithi122.stdout:Discarding device blocks: 1024/65536 done 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout:Creating filesystem with 65536 1k blocks and 16384 inodes 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout:Filesystem UUID: 3e6af7d0-32d0-4ab0-b9f8-e93f48d2c27a 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout:Superblock backups stored on blocks: 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout: 8193, 24577, 40961, 57345 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout: 2024-04-24T16:00:44.373 INFO:tasks.workunit.client.0.smithi122.stdout:Allocating group tables: 0/8 done 2024-04-24T16:00:44.385 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:44.385 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:44.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:44.385 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:44.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.067152937000003, 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.066001258999997, 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:44.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:44.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.393 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.050981475, 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:44.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:44.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:44.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:44.398 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.030049136999999, 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:44.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.029427183000003, 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:44.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.001442662000002, 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:44.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:44.410 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.988126972000003, 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.975415171000002, 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:44.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:44.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.916826266000001, 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:44.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.897390809999997, 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:44.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:44.427 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.884910828000002, 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.75747578, 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:44.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:44.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:44.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:44.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.756789032999997, 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.437 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.755634559000001, 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:44.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.751987133999997, 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.447 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:44.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:44.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.732027043000002, 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:44.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:44.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.730619134000001, 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.729983085000001, 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:44.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:44.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:44.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 51.546127575, 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:44.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:44.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.907383987999999, 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:44.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:44.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:44.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:44.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.474 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:44.474 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:44.474 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:44.476 INFO:tasks.workunit.client.0.smithi122.stdout:Writing inode tables: 0/8 done 2024-04-24T16:00:44.476 INFO:tasks.workunit.client.0.smithi122.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2024-04-24T16:00:44.476 INFO:tasks.workunit.client.0.smithi122.stdout: 2024-04-24T16:00:44.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:44.477 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ mkdir /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mount 2024-04-24T16:00:44.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/mount 2024-04-24T16:00:44.479 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:44.479 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:44.479 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:44.478+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:44.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:44.505 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd37497@quiesce3 2024-04-24T16:00:44.519 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:44.540 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:44.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:44.576 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:44.578+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:44.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:44.628 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:44.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:44.680 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:44.682+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:44.778 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:44.781+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:44.819 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:44.819 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:44.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:44.834 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:00:44.840 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T16:00:44.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/tmp/tmp.P3d5b6KQ4E/mnt/test bs=1M count=1 oflag=direct 2024-04-24T16:00:44.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:44.846 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:44.846 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.847 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:44.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:44.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:44.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/tmp/tmp.P3d5b6KQ4E/mnt/test bs=1M count=1 oflag=direct 2024-04-24T16:00:44.865 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:44.865 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:44.865 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281452 s, 373 MB/s 2024-04-24T16:00:44.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo umount /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:44.866 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:44.866 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:44.867 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.867 INFO:tasks.workunit.client.0.smithi122.stderr:++ which umount 2024-04-24T16:00:44.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/umount 2024-04-24T16:00:44.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:44.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:44.891 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:44.893+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 40075 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=40075 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:44.899 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:44.900 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:44.900 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:44.901 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:44.902 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:44.902 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:44.979 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:44.980 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:44.980 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:44.982+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:44.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:45.081 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:45.083+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:45.173 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.173 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.181 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:45.184+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:45.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:45.282 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:45.285+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:45.383 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:45.384+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:45.474 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.474 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:45.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^40075\b' 2024-04-24T16:00:45.478 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^40075\b' 2024-04-24T16:00:45.483 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:45.486+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:45.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:45.521 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:45.521 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:45.521 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *40075\b' 2024-04-24T16:00:45.521 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *40075\b' 2024-04-24T16:00:45.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:45.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:45.584 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:45.586+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:45.620 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.621 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:45.685 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:45.688+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:45.753 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.754 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:45.806 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:45.809+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:45.886 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:45.888+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:45.888 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:45.888 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:45.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:45.987 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:45.989+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:46.028 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:46.028 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat /tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:46.028 INFO:tasks.workunit.client.0.smithi122.stdout:2024-04-24T16:00:44.190+0000 7f2af3198180 0 ceph version 17.2.7-915-g330067db (330067dbce47655d1f762ef14d546d2fde16a6a9) quincy (stable), process rbd-nbd, pid 40066 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stdout:2024-04-24T16:00:44.190+0000 7f2af3198180 0 pidfile_write: ignore empty --pid-file 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stdout:2024-04-24T16:00:44.327+0000 7f2aebafd700 0 -- 172.21.15.122:0/4176599678 >> [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] conn(0x7f2ad0008740 msgr2=0x7f2ad000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stdout:2024-04-24T16:00:44.349+0000 7f2aebafd700 0 -- 172.21.15.122:0/4176599678 >> [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] conn(0x7f2acc03b800 msgr2=0x7f2acc03dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stdout:2024-04-24T16:00:44.966+0000 7f2acb7fe700 0 rbd-nbd: disconnect request received 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:46.029 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'quiesce failed' /tmp/tmp.P3d5b6KQ4E/rbd-nbd.log 2024-04-24T16:00:46.030 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:46.030 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd37497 2024-04-24T16:00:46.030 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:46.030 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:46.031 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:46.031 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:46.033 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:46.034 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:46.034 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd37497 2024-04-24T16:00:46.061 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.061 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:46.115 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:46.118+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:46.143 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 7271884c-5d6a-4a86-90dd-884e0a5df4ff' 2024-04-24T16:00:46.143 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV COOKIE 2024-04-24T16:00:46.143 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:46.144 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:46.144 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:46.144 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:46.144 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:46.189 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:46.191+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:46.197 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.197 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40209 2024-04-24T16:00:46.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40209 2024-04-24T16:00:46.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40209 -C rbd-nbd 2024-04-24T16:00:46.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:46.208 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:46.208 INFO:tasks.workunit.client.0.smithi122.stdout: 40209 ? 00:00:00 rbd-nbd 2024-04-24T16:00:46.208 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:46.208 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:46.209 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:46.209 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:46.209 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mount 2024-04-24T16:00:46.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/mount 2024-04-24T16:00:46.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:46.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:46.252 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.252 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:46.252 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:46.253 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:46.253 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:46.255 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:46.255 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:46.255 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.291 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:46.294+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:46.371 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.372 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:46.391 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:46.393+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:46.491 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:46.493+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:46.546 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.546 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:46.592 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:46.594+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:46.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:46.679 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.679 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:46.692 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:46.694+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:46.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:46.793 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:46.796+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:46.803 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:46.813 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:46.813 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:46.818 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-04-24T16:00:46.818 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:46.818 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:46.819 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:46.819 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:46.819 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:46.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:46.824 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:46.847 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:46.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T16:00:46.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.878 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.879 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:46.879 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:46.879 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:46.879 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:46.881 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:46.881 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:46.881 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.893 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:46.896+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:46.915 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: could not validate attach request 2024-04-24T16:00:46.915 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-04-24T16:00:46.915 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: must specify --cookie or --force to proceed 2024-04-24T16:00:46.917 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:46.917 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n 7271884c-5d6a-4a86-90dd-884e0a5df4ff ']' 2024-04-24T16:00:46.917 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 7271884c-5d6a-4a86-90dd-884e0a5df4ff rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.917 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:46.918 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:46.918 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:46.918 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:46.920 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:46.920 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:46.920 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 7271884c-5d6a-4a86-90dd-884e0a5df4ff rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:46.966 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:46.967 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:46.967 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:46.967 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:46.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:46.994 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:46.997+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:47.029 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-04-24T16:00:47.034 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:47.034 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:47.034 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:47.034 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:47.034 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:47.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40291 2024-04-24T16:00:47.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40291 2024-04-24T16:00:47.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40291 -C rbd-nbd 2024-04-24T16:00:47.095 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:47.097+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:47.099 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:47.099 INFO:tasks.workunit.client.0.smithi122.stdout: 40291 ? 00:00:00 rbd-nbd 2024-04-24T16:00:47.100 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T16:00:47.100 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:47.100 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:47.101 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:47.101 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:47.102 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:47.103 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:47.103 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T16:00:47.120 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:47.120 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:47.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:47.216 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:47.218+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:47.297 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:47.298+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:47.312 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.995049275, 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.313 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:47.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.993897597, 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.978877812999997, 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.323 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:47.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.327 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.957945475000002, 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:47.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:47.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:47.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.957323520999999, 2024-04-24T16:00:47.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:47.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:47.337 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.929338999999999, 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:47.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:47.342 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.91602331, 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:47.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:47.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:47.347 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.903311508999998, 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:47.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:47.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:47.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:47.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.844722603999998, 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.354 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:47.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:47.359 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.825287148000001, 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:47.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.812807165999999, 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:47.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.368 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.685372117999997, 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:47.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:47.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:47.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.684685371, 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.683530896999997, 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.679883472, 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.659923380999999, 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.658515471999998, 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.657879422999997, 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 54.474023913000003, 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.835280326000003, 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.425 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:47.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:47.427 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:47.429+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:47.528 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:47.530+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:47.613 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:47.613 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:47.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:47.628 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:47.630+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:47.665 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:47.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:47.665 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T16:00:47.665 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:00:47.666 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:00:47.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T16:00:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-04-24T16:00:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:47.671 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:47.671 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:47.728 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:47.730+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n 7271884c-5d6a-4a86-90dd-884e0a5df4ff ']' 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 7271884c-5d6a-4a86-90dd-884e0a5df4ff rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:47.729 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:47.730 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:47.730 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:47.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:47.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:47.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 7271884c-5d6a-4a86-90dd-884e0a5df4ff rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:47.763 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:47.763 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:47.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:47.796 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:47.796 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T16:00:47.796 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T16:00:47.840 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:47.842+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:47.841 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-04-24T16:00:47.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:47.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:47.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:47.846 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:47.846 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:47.905 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40353 2024-04-24T16:00:47.906 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40353 2024-04-24T16:00:47.906 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40353 -C rbd-nbd 2024-04-24T16:00:47.911 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:47.911 INFO:tasks.workunit.client.0.smithi122.stdout: 40353 ? 00:00:00 rbd-nbd 2024-04-24T16:00:47.911 INFO:tasks.workunit.client.0.smithi122.stderr:+ ls /tmp/tmp.P3d5b6KQ4E/mnt/ 2024-04-24T16:00:47.913 INFO:tasks.workunit.client.0.smithi122.stdout:lost+found 2024-04-24T16:00:47.913 INFO:tasks.workunit.client.0.smithi122.stdout:test 2024-04-24T16:00:47.913 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/tmp/tmp.P3d5b6KQ4E/mnt/test of=/dev/null bs=1M count=1 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643362 s, 163 MB/s 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.P3d5b6KQ4E/data of=/tmp/tmp.P3d5b6KQ4E/mnt/test1 bs=1M count=1 oflag=direct 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:47.921 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:47.922 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:47.922 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-04-24T16:00:47.924 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-04-24T16:00:47.924 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:47.924 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.P3d5b6KQ4E/data of=/tmp/tmp.P3d5b6KQ4E/mnt/test1 bs=1M count=1 oflag=direct 2024-04-24T16:00:47.929 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:47.931+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:47.952 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-04-24T16:00:47.952 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-04-24T16:00:47.952 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183073 s, 57.3 MB/s 2024-04-24T16:00:47.954 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo umount /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:47.954 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:47.954 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:47.955 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:47.955 INFO:tasks.workunit.client.0.smithi122.stderr:++ which umount 2024-04-24T16:00:47.957 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/umount 2024-04-24T16:00:47.957 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:47.957 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:00:47.966 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:47.966 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.660636768000003, 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:47.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:47.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:47.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:47.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:47.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.651016704, 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.630616136999997, 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.980 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:47.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:47.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.617684335, 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:47.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.617650584000003, 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:47.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:47.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:47.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:47.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.616128173999996, 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:47.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:47.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:47.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:47.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.610717057999999, 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:48.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.602968539999999, 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:48.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:48.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.600838517, 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:48.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.585135559999998, 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.015 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:48.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:48.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.585107573000002, 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.568336496999997, 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.029 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:48.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:48.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.562992973, 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.038 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:48.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.550782728000001, 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:48.048 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.543347562999998, 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:48.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.053 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.359091663999997, 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.354855737000001, 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 59.348015971000002, 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:48.063 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:48.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.593959553000001, 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:00:48.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:00:48.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:00:48.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:00:48.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.591734889000001, 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:48.073 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:00:48.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:00:48.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.079 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:48.080 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:48.083+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:48.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:48.082 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 40353 2024-04-24T16:00:48.082 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=40353 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:48.083 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:48.181 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:48.183+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:48.270 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:48.270 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:48.270 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:48.270 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:48.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:48.283 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:48.285+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:48.384 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:48.386+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:48.462 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:48.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:48.463 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:48.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:48.518 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:48.520+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:48.569 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:48.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^40353\b' 2024-04-24T16:00:48.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^40353\b' 2024-04-24T16:00:48.585 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:48.587+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:48.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:48.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:48.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *40353\b' 2024-04-24T16:00:48.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *40353\b' 2024-04-24T16:00:48.614 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:48.614 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:48.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:48.614 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.236742176, 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:00:48.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.235878829000001, 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:00:48.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.232496564000002, 2024-04-24T16:00:48.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.182301226, 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:00:48.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:00:48.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.177267796999999, 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.173313049000001, 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:00:48.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.148382882, 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:48.641 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.233257988999998, 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:76 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'296, mlcod=80'294)", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:48.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.284664856, 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055479819999999999, 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331215+0000", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331221+0000", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331233+0000", 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1596999999999999e-05 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331240+0000", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8920000000000003e-06 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331282+0000", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1681999999999998e-05 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331301+0000", 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9613999999999999e-05 2024-04-24T16:00:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336737+0000", 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054352849999999998 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336764+0000", 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7348e-05 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.649 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:77 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'297, mlcod=80'294)", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2837629100000001, 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046695269999999997, 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332117+0000", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332125+0000", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332139+0000", 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3392e-05 2024-04-24T16:00:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332146+0000", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3050000000000004e-06 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332193+0000", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6969999999999999e-05 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332216+0000", 2024-04-24T16:00:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3156e-05 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336766+0000", 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045501680000000003 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336787+0000", 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1287000000000001e-05 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "pg_update_log_missing(3.0 epoch 80/67 rep_tid 482 entries 80'298 (0'0) error 3:372fdc5e:::rbd_header.129ad8b6ae3f:head by client.5285.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 80'294)", 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.28296688, 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038821250000000002, 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332913+0000", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332923+0000", 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9159999999999996e-06 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332940+0000", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6986e-05 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332949+0000", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8289999999999997e-06 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333006+0000", 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6969999999999998e-05 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336796+0000", 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003790093 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:79 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'299, mlcod=80'294)", 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2824428609999998, 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003383718, 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333437+0000", 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333444+0000", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4290000000000004e-06 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333455+0000", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0828e-05 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333461+0000", 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4639999999999997e-06 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333503+0000", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2086999999999999e-05 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333523+0000", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0094000000000001e-05 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336797+0000", 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032740569999999999 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336822+0000", 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4691000000000001e-05 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:82 3.0 e80/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 80'301, mlcod=80'294)", 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2816366180000003, 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003048916, 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.661 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334243+0000", 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334250+0000", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1530000000000002e-06 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334260+0000", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8810000000000008e-06 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334264+0000", 2024-04-24T16:00:48.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4410000000000003e-06 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334305+0000", 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0880999999999997e-05 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334336+0000", 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1554999999999999e-05 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337270+0000", 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002933546 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337293+0000", 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.315e-05 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:84 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'303, mlcod=80'299)", 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2756235980000001, 2024-04-24T16:00:48.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044460719999999997, 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340256+0000", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340263+0000", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340274+0000", 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0509000000000001e-05 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340281+0000", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7290000000000003e-06 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340339+0000", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8041000000000002e-05 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340361+0000", 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1892e-05 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344658+0000", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042975230000000001 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344703+0000", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:85 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'305, mlcod=80'299)", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.274847598, 2024-04-24T16:00:48.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042239729999999998, 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341038+0000", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9869999999999996e-06 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341054+0000", 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5784e-05 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341071+0000", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341122+0000", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.109e-05 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341142+0000", 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9956999999999999e-05 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345218+0000", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040761019999999999 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345257+0000", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8852e-05 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:101 3.0 e80/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 80'318, mlcod=80'314)", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2593803389999998, 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041012970000000003, 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356507+0000", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0140000000000001e-06 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356517+0000", 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.005e-05 2024-04-24T16:00:48.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356523+0000", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4489999999999996e-06 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356566+0000", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2168e-05 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356582+0000", 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6711e-05 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360572+0000", 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039900580000000003 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360602+0000", 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9408e-05 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:102 3.0 e80/67 3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 80'320, mlcod=80'314)", 2024-04-24T16:00:48.677 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2536933980000002, 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044501560000000003, 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362193+0000", 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0310000000000004e-06 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362203+0000", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0345e-05 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362208+0000", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9300000000000002e-06 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362246+0000", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8188000000000001e-05 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362259+0000", 2024-04-24T16:00:48.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3057999999999999e-05 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366606+0000", 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043471830000000001 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366638+0000", 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1063999999999999e-05 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:107 3.0 e80/67 3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 80'325, mlcod=80'320)", 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.2458307189999998, 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045367460000000004, 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370049+0000", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370055+0000", 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2560000000000002e-06 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370064+0000", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1910000000000006e-06 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370068+0000", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.267e-06 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370105+0000", 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6418999999999998e-05 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370130+0000", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5082000000000001e-05 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374558+0000", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004428871 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374587+0000", 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.832e-05 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:139 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'343, mlcod=80'339)", 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.216856409, 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001640063, 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399023+0000", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399118+0000", 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4485000000000001e-05 2024-04-24T16:00:48.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399139+0000", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0815e-05 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399145+0000", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4969999999999998e-06 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399181+0000", 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5942000000000002e-05 2024-04-24T16:00:48.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399204+0000", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2826000000000001e-05 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400627+0000", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014233449999999999 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400664+0000", 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7036999999999998e-05 2024-04-24T16:00:48.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:140 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'344, mlcod=80'339)", 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.216033328, 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021562809999999999, 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:48.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:48.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399951+0000", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000104269 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399966+0000", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5350999999999999e-05 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399973+0000", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2750000000000001e-06 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400011+0000", 2024-04-24T16:00:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8074000000000002e-05 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400032+0000", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0789e-05 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.401963+0000", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019310410000000001 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.402004+0000", 2024-04-24T16:00:48.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1174999999999998e-05 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:48.694 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:48.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:48.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:48.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:48.696 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:48.698+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:48.796 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:48.798+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:48.831 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:48.831 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:48.831 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:48.832 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:48.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:48.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:48.896 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:48.899+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:48.965 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:48.965 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:48.965 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:48.965 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:48.966 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:48.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:48.997 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:48.999+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:49.051 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:49.071 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:49.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:49.098 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:49.100+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:49.110 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:49.119 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n 7271884c-5d6a-4a86-90dd-884e0a5df4ff ']' 2024-04-24T16:00:49.119 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd37497 2024-04-24T16:00:49.119 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:49.119 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:49.120 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:49.121 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:49.123 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:49.123 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:49.123 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd37497 2024-04-24T16:00:49.130 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:49.140 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.140 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:49.199 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:49.201+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:49.241 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 abc de' 2024-04-24T16:00:49.241 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV ANOTHER_COOKIE 2024-04-24T16:00:49.241 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:49.241 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:49.241 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:49.242 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:49.242 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:49.278 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.278 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:49.298 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40438 2024-04-24T16:00:49.299 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40438 2024-04-24T16:00:49.299 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40438 -C rbd-nbd 2024-04-24T16:00:49.300 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:49.302+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:49.304 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:49.304 INFO:tasks.workunit.client.0.smithi122.stdout: 40438 ? 00:00:00 rbd-nbd 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 'abc de' = 'abc de' 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 40438 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=40438 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:49.305 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:49.306 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:49.306 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:49.307 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:49.307 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:49.307 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T16:00:49.389 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:49.390 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^40438\b' 2024-04-24T16:00:49.390 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^40438\b' 2024-04-24T16:00:49.401 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:49.403+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:49.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:49.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:49.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *40438\b' 2024-04-24T16:00:49.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *40438\b' 2024-04-24T16:00:49.435 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:49.435 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:49.451 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.452 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:49.502 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:49.505+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:49.603 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:49.606+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:49.625 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.625 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:49.704 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:49.707+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:49.799 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.800 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.805 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:49.808+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:49.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:49.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:00:49.906 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:49.908+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:49.932 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:49.932 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:49.936 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:49.937 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/testrbdnbd37497 2024-04-24T16:00:49.937 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T16:00:49.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:49.977 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=4 2024-04-24T16:00:49.977 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd37497 2024-04-24T16:00:49.977 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:49.977 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:49.978 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:49.978 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:49.980 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:49.980 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:49.980 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd37497 2024-04-24T16:00:50.009 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:50.009+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:50.068 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.069 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 fd55d9e2-681c-4ce8-8fa8-fdfa3e3e6720' 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV COOKIE 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:50.091 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:50.120 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:50.123+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:50.149 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40533 2024-04-24T16:00:50.150 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40533 2024-04-24T16:00:50.150 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40533 -C rbd-nbd 2024-04-24T16:00:50.155 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:50.155 INFO:tasks.workunit.client.0.smithi122.stdout: 40533 ? 00:00:00 rbd-nbd 2024-04-24T16:00:50.155 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach rbd/testrbdnbd37497 --snap-id 4 --device-type nbd 2024-04-24T16:00:50.156 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:50.156 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:50.156 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:50.156 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:50.158 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:50.158 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:50.158 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd37497 --snap-id 4 --device-type nbd 2024-04-24T16:00:50.209 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:50.211+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:50.219 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.220 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:50.310 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:50.312+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:50.410 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:50.412+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:50.413 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.413 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:50.511 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:50.514+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:50.606 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.607 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.612 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:50.614+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:50.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:50.714 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:50.715+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:50.728 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:50.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:50.787 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:50.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:50.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:50.788 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:50.801 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.801 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:50.814 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:50.816+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:50.823 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: could not validate attach request 2024-04-24T16:00:50.823 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-04-24T16:00:50.823 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: must specify --cookie or --force to proceed 2024-04-24T16:00:50.826 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:50.826 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n fd55d9e2-681c-4ce8-8fa8-fdfa3e3e6720 ']' 2024-04-24T16:00:50.826 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie fd55d9e2-681c-4ce8-8fa8-fdfa3e3e6720 --snap-id 4 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:50.826 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:50.826 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:50.827 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:50.827 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:50.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:50.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:50.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie fd55d9e2-681c-4ce8-8fa8-fdfa3e3e6720 --snap-id 4 rbd/testrbdnbd37497 --device-type nbd 2024-04-24T16:00:50.846 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:00:50.915 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:50.917+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:50.948 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-04-24T16:00:50.952 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:50.953 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:50.953 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:50.953 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:50.953 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:50.956 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:50.956 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:50.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:51.014 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40607 2024-04-24T16:00:51.014 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40607 2024-04-24T16:00:51.014 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40607 -C rbd-nbd 2024-04-24T16:00:51.015 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:51.018+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:51.019 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:51.019 INFO:tasks.workunit.client.0.smithi122.stdout: 40607 ? 00:00:00 rbd-nbd 2024-04-24T16:00:51.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T16:00:51.020 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:51.020 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:51.020 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:51.020 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:51.022 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:51.022 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:51.022 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T16:00:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:51.116 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:51.119+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:51.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:51.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:51.217 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:51.220+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:51.261 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:51.261 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:51.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:51.318 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:51.321+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:51.325 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:51.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:51.364 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:51.385 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:51.406 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:51.415 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:51.415 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:51.415 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:51.415 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:51.416 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:51.419 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:51.422+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:51.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:51.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:51.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:51.428 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:51.428 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:51.428 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:51.428 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:51.428 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:51.520 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:51.523+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:51.568 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.190147818, 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:00:51.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:00:51.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:00:51.572 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.189284471000001, 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:00:51.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.185902206000002, 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:00:51.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.135706868, 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.130673438999999, 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:00:51.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.126718691000001, 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:00:51.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.101788524, 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:00:51.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:00:51.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.186663630999998, 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:76 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'296, mlcod=80'294)", 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2380704979999999, 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055479819999999999, 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331215+0000", 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331221+0000", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331233+0000", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1596999999999999e-05 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331240+0000", 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8920000000000003e-06 2024-04-24T16:00:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331282+0000", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1681999999999998e-05 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331301+0000", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9613999999999999e-05 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336737+0000", 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054352849999999998 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336764+0000", 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7348e-05 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:77 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'297, mlcod=80'294)", 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.237168552, 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046695269999999997, 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332117+0000", 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332125+0000", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332139+0000", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3392e-05 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332146+0000", 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3050000000000004e-06 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332193+0000", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6969999999999999e-05 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332216+0000", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3156e-05 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336766+0000", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045501680000000003 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336787+0000", 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1287000000000001e-05 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "pg_update_log_missing(3.0 epoch 80/67 rep_tid 482 entries 80'298 (0'0) error 3:372fdc5e:::rbd_header.129ad8b6ae3f:head by client.5285.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 80'294)", 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2363725219999999, 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038821250000000002, 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332913+0000", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332923+0000", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9159999999999996e-06 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332940+0000", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6986e-05 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332949+0000", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8289999999999997e-06 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333006+0000", 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6969999999999998e-05 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336796+0000", 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003790093 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:79 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'299, mlcod=80'294)", 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2358485029999997, 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003383718, 2024-04-24T16:00:51.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333437+0000", 2024-04-24T16:00:51.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333444+0000", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4290000000000004e-06 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333455+0000", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0828e-05 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333461+0000", 2024-04-24T16:00:51.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4639999999999997e-06 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333503+0000", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2086999999999999e-05 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333523+0000", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0094000000000001e-05 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336797+0000", 2024-04-24T16:00:51.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032740569999999999 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336822+0000", 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4691000000000001e-05 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:82 3.0 e80/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 80'301, mlcod=80'294)", 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2350422600000002, 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003048916, 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334243+0000", 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334250+0000", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1530000000000002e-06 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334260+0000", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8810000000000008e-06 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334264+0000", 2024-04-24T16:00:51.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4410000000000003e-06 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334305+0000", 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0880999999999997e-05 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334336+0000", 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1554999999999999e-05 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337270+0000", 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002933546 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337293+0000", 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.315e-05 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:84 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'303, mlcod=80'299)", 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.22902924, 2024-04-24T16:00:51.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044460719999999997, 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340256+0000", 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340263+0000", 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340274+0000", 2024-04-24T16:00:51.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0509000000000001e-05 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340281+0000", 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7290000000000003e-06 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340339+0000", 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8041000000000002e-05 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340361+0000", 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1892e-05 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344658+0000", 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042975230000000001 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344703+0000", 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:85 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'305, mlcod=80'299)", 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:51.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2282532399999999, 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042239729999999998, 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341038+0000", 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9869999999999996e-06 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341054+0000", 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5784e-05 2024-04-24T16:00:51.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341071+0000", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341122+0000", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.109e-05 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341142+0000", 2024-04-24T16:00:51.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9956999999999999e-05 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345218+0000", 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040761019999999999 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345257+0000", 2024-04-24T16:00:51.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8852e-05 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:101 3.0 e80/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 80'318, mlcod=80'314)", 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2127859809999997, 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041012970000000003, 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.632 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356507+0000", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0140000000000001e-06 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356517+0000", 2024-04-24T16:00:51.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.005e-05 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356523+0000", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4489999999999996e-06 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356566+0000", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2168e-05 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356582+0000", 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6711e-05 2024-04-24T16:00:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360572+0000", 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039900580000000003 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360602+0000", 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9408e-05 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:102 3.0 e80/67 3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 80'320, mlcod=80'314)", 2024-04-24T16:00:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.2070990400000001, 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044501560000000003, 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362193+0000", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0310000000000004e-06 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362203+0000", 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0345e-05 2024-04-24T16:00:51.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362208+0000", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9300000000000002e-06 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362246+0000", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8188000000000001e-05 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362259+0000", 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3057999999999999e-05 2024-04-24T16:00:51.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366606+0000", 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043471830000000001 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366638+0000", 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1063999999999999e-05 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:107 3.0 e80/67 3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 80'325, mlcod=80'320)", 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.1992363609999996, 2024-04-24T16:00:51.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045367460000000004, 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370049+0000", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370055+0000", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2560000000000002e-06 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370064+0000", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1910000000000006e-06 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370068+0000", 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.267e-06 2024-04-24T16:00:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370105+0000", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6418999999999998e-05 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370130+0000", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5082000000000001e-05 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374558+0000", 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004428871 2024-04-24T16:00:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374587+0000", 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.832e-05 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:139 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'343, mlcod=80'339)", 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.1702620509999999, 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001640063, 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399023+0000", 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399118+0000", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4485000000000001e-05 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399139+0000", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0815e-05 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399145+0000", 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4969999999999998e-06 2024-04-24T16:00:51.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399181+0000", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5942000000000002e-05 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399204+0000", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2826000000000001e-05 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400627+0000", 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014233449999999999 2024-04-24T16:00:51.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400664+0000", 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7036999999999998e-05 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:140 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'344, mlcod=80'339)", 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 7.1694389699999999, 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021562809999999999, 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:51.650 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399951+0000", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000104269 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399966+0000", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5350999999999999e-05 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399973+0000", 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2750000000000001e-06 2024-04-24T16:00:51.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400011+0000", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8074000000000002e-05 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400032+0000", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0789e-05 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.401963+0000", 2024-04-24T16:00:51.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019310410000000001 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.402004+0000", 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1174999999999998e-05 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:51.654 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:51.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:51.657 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:51.658+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:51.658 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config image set rbd/testrbdnbd37497 rbd_discard_granularity_bytes 4096 2024-04-24T16:00:51.706 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature enable rbd/testrbdnbd37497 journaling 2024-04-24T16:00:51.757 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:51.759+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:51.768 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:51.768 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:51.768 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:51.769 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:51.769 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:51.771 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:51.771 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:51.771 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:51.797 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:51.798 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:51.798 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:51.798 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:51.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:51.857 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:51.860+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:51.872 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T16:00:51.947 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=40712 2024-04-24T16:00:51.948 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 40712 2024-04-24T16:00:51.948 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 40712 -C rbd-nbd 2024-04-24T16:00:51.952 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:51.952 INFO:tasks.workunit.client.0.smithi122.stdout: 40712 ? 00:00:00 rbd-nbd 2024-04-24T16:00:51.953 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-04-24T16:00:51.953 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:51.953 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:51.953 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:51.954 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:51.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:51.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:51.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-04-24T16:00:51.958 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:51.960+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:51.966 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T16:00:51.967 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:51.967 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:51.967 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:51.968 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:51.969 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:51.969 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:51.970 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T16:00:51.972 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:51.972 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:51.973 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:51.973 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:51.973 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:51.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:51.986 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-04-24T16:00:51.986 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:51.986 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:51.987 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:51.987 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:51.989 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:51.989 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:51.989 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-04-24T16:00:52.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-04-24T16:00:52.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:52.005 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:52.005 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:52.005 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:52.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:52.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:52.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-04-24T16:00:52.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-04-24T16:00:52.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:52.018 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:52.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:52.019 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:52.020 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:52.020 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:52.020 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-04-24T16:00:52.036 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-04-24T16:00:52.036 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:52.036 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:52.036 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:52.036 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:52.038 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:52.038 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:52.038 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-04-24T16:00:52.053 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=0 2024-04-24T16:00:52.054 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:52.054 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:52.059 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:52.061+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:52.109 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:52.111+0000 7f29f77fe700 -1 JournalMetadata: 0x562134abe5a0 failed to locate client: INSPECT 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.803311837999999, 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:00:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:00:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:00:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:00:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.795345808, 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:00:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:00:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.795311613000003, 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:00:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.795292975000002, 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.787970238, 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:00:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:00:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.766311215000002, 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.762062581000002, 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:00:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:00:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.762028063000002, 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:00:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:00:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.731866959000001, 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:00:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:00:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.680772019999999, 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.627253508000003, 2024-04-24T16:00:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:00:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:00:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.503913347999998, 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.487043892999999, 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.478827467999999, 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:00:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:00:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.465287320000002, 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:00:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.463402651000003, 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:00:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:00:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:00:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.442794392000003, 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.432872891999999, 2024-04-24T16:00:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:00:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:00:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.428224634000003, 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.426730843000001, 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.200 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:52.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-04-24T16:00:52.202 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -lt 10 ']' 2024-04-24T16:00:52.202 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T16:00:52.202 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd37497 --verbose 2024-04-24T16:00:52.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:52.203 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:52.204+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:52.203 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:51.982814+0000" 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.002073+0000" 2024-04-24T16:00:52.204 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.033522+0000" 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:52.205 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.051509+0000" 2024-04-24T16:00:52.206 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:52.206 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-04-24T16:00:52.206 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-04-24T16:00:52.206 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:52.172+0000 7ff84a86f700 -1 JournalMetadata: 0x56057d4b5340 failed to locate client: INSPECT 2024-04-24T16:00:52.206 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-04-24T16:00:52.302 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:52.304+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:52.366 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:52.366 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:52.366 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:52.367 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:52.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:52.403 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:52.406+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:52.504 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:52.507+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:52.540 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:52.540 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:52.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:52.605 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:52.607+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:52.706 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:52.708+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.419090569000005, 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.724 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:00:52.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:00:52.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:00:52.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:00:52.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.417943782999998, 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:00:52.729 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.733 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.402091661, 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:52.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:00:52.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:00:52.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.379327029999999, 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.369208674000006, 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.341265558999993, 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:00:52.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:00:52.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:00:52.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.341247733000003, 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:52.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:00:52.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:00:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:00:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:00:52.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.331630335, 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:52.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:00:52.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:00:52.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:00:52.759 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.327901424000004, 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:00:52.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:00:52.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:00:52.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.302017360999997, 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.245795720999993, 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.768 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.245774611000002, 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:52.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:52.778 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.132899765999994, 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.123481995999995, 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:52.786 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.106965829999993, 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:52.790 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.106409413999998, 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:00:52.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.057179009999999, 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:00:52.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:00:52.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:00:52.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:00:52.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.045501182999999, 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.799 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.044153424000001, 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:00:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.036025843999994, 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:52.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:00:52.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:52.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:52.812 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:52.812 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:00:52.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T16:00:52.814 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:52.816+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:52.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:52.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:00:52.914 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:52.917+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:52.980 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:52.980 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:52.980 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:00:52.980 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:52.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:53.031 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:53.034+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:53.115 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:53.117+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:53.154 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:53.154 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:53.154 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:53.154 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:53.154 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:53.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:53.174 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:53.174 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:53.217 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:53.219+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.967223880999995, 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:53.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:53.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.965419300999997, 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.296 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:53.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:53.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.965387933000002, 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:53.301 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:53.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:53.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.941664207000002, 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.913806468999994, 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.310 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:53.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:53.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:53.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:53.314 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.909176963999997, 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:53.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:53.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.902369421000003, 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:53.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:53.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:53.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.871829194, 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.856189513999993, 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:53.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:53.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:53.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.817014264999997, 2024-04-24T16:00:53.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:53.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:53.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.804396144999998, 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.340 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.682000008000003, 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:53.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.668470799999994, 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.345 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:53.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:53.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:53.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.657736471999996, 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:53.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:53.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:53.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.643440329000001, 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:53.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:53.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:53.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:53.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.611213237000001, 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:53.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:53.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:53.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.605117395999997, 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.601662838999999, 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:53.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.566218718000002, 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:53.375 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:53.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:53.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:53.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 57.514758952999998, 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:53.379 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:53.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:53.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:53.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:53.383 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:53.385 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:53.387+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:53.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:53.436 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:53.438+0000 7fb67595c700 -1 JournalMetadata: 0x56543f63b9d0 failed to locate client: INSPECT 2024-04-24T16:00:53.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-04-24T16:00:53.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=1 2024-04-24T16:00:53.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1 -lt 10 ']' 2024-04-24T16:00:53.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T16:00:53.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd37497 --verbose 2024-04-24T16:00:53.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:53.487 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:53.489+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:53.496 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:51.982814+0000" 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.002073+0000" 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:53.497 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.033522+0000" 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:53.498 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.051509+0000" 2024-04-24T16:00:53.499 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:53.499 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-04-24T16:00:53.499 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-04-24T16:00:53.499 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:53.501+0000 7fc49dffb700 -1 JournalMetadata: 0x56345ea55a00 failed to locate client: INSPECT 2024-04-24T16:00:53.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-04-24T16:00:53.537 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:53.538 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:53.538 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:53.538 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:53.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:53.588 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:53.591+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:53.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:53.623 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:53.644 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:53.664 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:53.684 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:53.689 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:53.691+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:53.692 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:53.693 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:53.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:53.705 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:53.790 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:53.792+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:53.845 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:53.845 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T16:00:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:00:53.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:53.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T16:00:53.914 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:53.916+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:53.991 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:53.993+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:54.011 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.011 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:54.108 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:54.110+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:54.194 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:54.196+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:54.203 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.203 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:54.295 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:54.297+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:54.394 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.394 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.395 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:54.397+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:54.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:54.496 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:54.498+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:54.506 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:54.506 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:54.566 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:54.568+0000 7fdf6d0cb700 -1 JournalMetadata: 0x55685b2aa7d0 failed to locate client: INSPECT 2024-04-24T16:00:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-04-24T16:00:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=2 2024-04-24T16:00:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 2 -lt 10 ']' 2024-04-24T16:00:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T16:00:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd37497 --verbose 2024-04-24T16:00:54.589 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.590 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.597 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:54.599+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:54.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:51.982814+0000" 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:54.629 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.002073+0000" 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.033522+0000" 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T16:00:54.630 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.051509+0000" 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-04-24T16:00:54.631 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-04-24T16:00:54.634 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:54.636+0000 7fd84e1b4700 -1 JournalMetadata: 0x56178eb7f240 failed to locate client: INSPECT 2024-04-24T16:00:54.636 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-04-24T16:00:54.698 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:54.701+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:54.741 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.741 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:54.799 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:54.802+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T16:00:54.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T16:00:54.895 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:54.895 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:54.900 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:54.903+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:54.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:55.001 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:55.004+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:55.045 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.045 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:55.102 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:55.105+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:55.176 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.177 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:55.202 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:55.206+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:55.303 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:55.306+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:55.309 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.309 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:55.404 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:55.407+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:55.482 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.483 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:55.533 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:55.536+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:55.606 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:55.609+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:55.638 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:55.638 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:55.657 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.657 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:55.701 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:55.702+0000 7f11d4b8d700 -1 JournalMetadata: 0x5620680aac00 failed to locate client: INSPECT 2024-04-24T16:00:55.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-04-24T16:00:55.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=3 2024-04-24T16:00:55.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 3 -lt 10 ']' 2024-04-24T16:00:55.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T16:00:55.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd37497 --verbose 2024-04-24T16:00:55.707 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:55.710+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:55.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:51.982814+0000" 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T16:00:55.759 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.002073+0000" 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.033522+0000" 2024-04-24T16:00:55.760 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.051509+0000" 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-04-24T16:00:55.761 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-04-24T16:00:55.766 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:55.769+0000 7f79908b2700 -1 JournalMetadata: 0x560fcee257d0 failed to locate client: INSPECT 2024-04-24T16:00:55.769 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-04-24T16:00:55.808 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:55.810+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:55.833 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:55.833 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:55.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:55.860 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T16:00:55.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:00:55.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T16:00:55.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:55.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:55.926 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:55.929+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:55.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:55.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:56.007 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:56.007 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:56.009 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:56.010 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:56.012+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:56.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:56.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:56.031 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:56.032 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:56.111 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:56.113+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:56.181 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:56.181 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:00:56.181 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:00:56.181 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:00:56.181 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:56.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:56.212 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:56.215+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:56.313 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:56.316+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.357 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 68.034515378999998, 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:00:56.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:00:56.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 68.032710799, 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:56.367 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 68.032679431000005, 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:00:56.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:56.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 68.008955705000005, 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.981097966999997, 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:00:56.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.976468462, 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:56.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:00:56.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:00:56.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:00:56.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.969660919000006, 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:00:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:00:56.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.939120692000003, 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:00:56.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.923481011999996, 2024-04-24T16:00:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.396 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.884305763, 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:00:56.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:56.401 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.871687643000001, 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.749291506000006, 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:00:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.735762297999997, 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:00:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.725027969999999, 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:00:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:00:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:00:56.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.418 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.710731827000004, 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:00:56.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:00:56.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.423 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.678504735000004, 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.424 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.428 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.672408894, 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:00:56.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:00:56.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:00:56.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:00:56.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.668954337000002, 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.633510216000005, 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:00:56.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:00:56.441 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.582050451000001, 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:56.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:00:56.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.445 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:00:56.446 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:00:56.446 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:56.447 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:56.449+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:56.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:56.548 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:56.551+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:56.594 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:56.595 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:56.595 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:56.595 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:56.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:56.649 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:56.652+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:56.749 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:56.752+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:56.771 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:56.771 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:56.787 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:56.788 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:56.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:00:56.831 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:56.833+0000 7f5d03fff700 -1 JournalMetadata: 0x55f5276ca780 failed to locate client: INSPECT 2024-04-24T16:00:56.835 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-04-24T16:00:56.835 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=4 2024-04-24T16:00:56.836 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -lt 10 ']' 2024-04-24T16:00:56.836 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T16:00:56.836 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd37497 --verbose 2024-04-24T16:00:56.851 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:56.853+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:00:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T16:00:56.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:51.982814+0000" 2024-04-24T16:00:56.892 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.002073+0000" 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:56.893 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.033522+0000" 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-04-24T16:00:52.051509+0000" 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-04-24T16:00:56.894 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-04-24T16:00:56.895 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:56.898+0000 7fb8ca7fc700 -1 JournalMetadata: 0x55c252334210 failed to locate client: INSPECT 2024-04-24T16:00:56.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-04-24T16:00:56.952 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:56.954+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:56.996 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.679114893000005, 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:56.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.677963215000005, 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.662943431000002, 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:00:57.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.642011092999994, 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:00:57.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.641389138999997, 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:00:57.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:00:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.613404618000004, 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.017 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.600088928000005, 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:00:57.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:00:57.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.587377126999996, 2024-04-24T16:00:57.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:57.029 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.528788222000003, 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.509352766000006, 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:57.037 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.496872784000004, 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:00:57.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:00:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.369437735999995, 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:00:57.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:00:57.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:57.046 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.368750989000006, 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:00:57.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:00:57.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.051 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.367596515000002, 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:00:57.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:00:57.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:00:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:00:57.055 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.363949090000006, 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:00:57.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.059 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.343988999000004, 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:00:57.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:00:57.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.064 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.342581089999996, 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.341945041000002, 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:00:57.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:00:57.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:00:57.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:00:57.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:00:57.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 64.158089531000002, 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:00:57.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:00:57.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:00:57.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:00:57.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.519345944000001, 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:00:57.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:00:57.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:00:57.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.082 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:57.083 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:57.086+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:57.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:00:57.185 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:57.188+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:57.223 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:57.223 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:57.223 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:57.223 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:57.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:00:57.320 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:57.322+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:57.387 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:57.391+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:57.413 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:57.413 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:57.413 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:57.413 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:57.413 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:57.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:00:57.508 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:57.511+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:57.591 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:57.593+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:57.607 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:00:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:00:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.302969607999998, 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:00:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:00:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.293349543999994, 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:00:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.272948976999999, 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:00:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.260017175000002, 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:00:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.259983423999998, 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.258461014000005, 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:00:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:00:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:00:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.253049898, 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.245301380000001, 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:00:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.243171356999994, 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.227468400000006, 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:00:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:00:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.227440412999997, 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:00:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.210669336999999, 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:00:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:00:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.205325813000002, 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.193115567999996, 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:00:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.185680403000006, 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:00:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 69.001424503999999, 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.997188577000003, 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:00:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:00:57.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 68.990348811000004, 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:00:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:00:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 33.236292392999999, 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:57.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 33.234067729000003, 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:00:57.702 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:00:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:00:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:00:57.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:00:57.709 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:57.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:00:57.711 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:57.712+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:57.811 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:57.814+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:57.873 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T16:00:57.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T16:00:57.901 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd37497 2024-04-24T16:00:57.901 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T16:00:57.904 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:00:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:57.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:00:57.918 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:57.920+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:58.012 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:58.015+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:58.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:00:58.113 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:58.115+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:58.120 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:00:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:00:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:00:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:00:58.120 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:00:58.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:00:58.165 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:00:58.167+0000 7fe715eb9700 -1 JournalMetadata: 0x5623533da4d0 failed to locate client: INSPECT 2024-04-24T16:00:58.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=0 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=5 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 5 -lt 10 ']' 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -gt 0 ']' 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 40712 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=40712 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:58.171 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:58.172 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:58.172 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:58.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:58.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:58.174 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:00:58.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:00:58.214 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:58.217+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:58.229 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:00:58.250 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:00:58.270 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:00:58.293 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:00:58.313 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:00:58.314 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:58.318+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:00:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:00:58.321 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:58.321 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:58.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:00:58.415 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:58.418+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:58.461 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:58.461 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:58.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:00:58.554 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:58.557+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:58.617 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:58.620+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:58.633 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:58.634 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:58.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:00:58.717 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:58.721+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:58.776 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:00:58.776 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^40712\b' 2024-04-24T16:00:58.776 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^40712\b' 2024-04-24T16:00:58.807 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:58.807 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:58.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:00:58.818 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:58.820+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:58.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:58.830 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:00:58.830 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *40712\b' 2024-04-24T16:00:58.830 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *40712\b' 2024-04-24T16:00:58.836 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:00:58.836 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:00:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T16:00:58.889 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T16:00:58.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:58.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T16:00:58.919 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:58.922+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:58.942 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:58.943 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:58.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:00:59.020 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:59.023+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:59.117 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.117 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.120 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:59.123+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:59.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:00:59.222 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:00:59.225+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:00:59.249 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.250 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:00:59.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:59.295 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:59.308 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-04-24T16:00:58.871304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231919403,"num_objects":123,"num_object_clones":19,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":14,"num_read":6177,"num_read_kb":174537,"num_write":994,"num_write_kb":273494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":97059600,"num_keys_recovered":26,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1019,"ondisk_log_size":1019,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2247036,"kb_used_data":446508,"kb_used_omap":9,"kb_used_meta":1800438,"kb_avail":563984004,"statfs":{"total":579820584960,"available":577519620096,"internally_reserved":0,"allocated":457224192,"data_stored":466951258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843648528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6768,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":863,"num_read_kb":1682,"num_write":31,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015501"},"pg_stats":[{"pgid":"1.5","version":"83'6","reported_seq":131,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678349+0000","last_change":"2024-04-24T16:00:22.519701+0000","last_active":"2024-04-24T16:00:55.678349+0000","last_peered":"2024-04-24T16:00:55.678349+0000","last_clean":"2024-04-24T16:00:55.678349+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:00:55.678349+0000","last_undegraded":"2024-04-24T16:00:55.678349+0000","last_fullsized":"2024-04-24T16:00:55.678349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858785+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:54.858785+0000","last_peered":"2024-04-24T16:00:54.858785+0000","last_clean":"2024-04-24T16:00:54.858785+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:54.858785+0000","last_undegraded":"2024-04-24T16:00:54.858785+0000","last_fullsized":"2024-04-24T16:00:54.858785+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":141,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856898+0000","last_change":"2024-04-24T16:00:31.523686+0000","last_active":"2024-04-24T16:00:54.856898+0000","last_peered":"2024-04-24T16:00:54.856898+0000","last_clean":"2024-04-24T16:00:54.856898+0000","last_became_active":"2024-04-24T16:00:31.414295+0000","last_became_peered":"2024-04-24T16:00:31.414295+0000","last_unstale":"2024-04-24T16:00:54.856898+0000","last_undegraded":"2024-04-24T16:00:54.856898+0000","last_fullsized":"2024-04-24T16:00:54.856898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:31.661486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":11,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678359+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:55.678359+0000","last_peered":"2024-04-24T16:00:55.678359+0000","last_clean":"2024-04-24T16:00:55.678359+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:55.678359+0000","last_undegraded":"2024-04-24T16:00:55.678359+0000","last_fullsized":"2024-04-24T16:00:55.678359+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858939+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:54.858939+0000","last_peered":"2024-04-24T16:00:54.858939+0000","last_clean":"2024-04-24T16:00:54.858939+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:54.858939+0000","last_undegraded":"2024-04-24T16:00:54.858939+0000","last_fullsized":"2024-04-24T16:00:54.858939+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"81'6","reported_seq":111,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856741+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:54.856741+0000","last_peered":"2024-04-24T16:00:54.856741+0000","last_clean":"2024-04-24T16:00:54.856741+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:54.856741+0000","last_undegraded":"2024-04-24T16:00:54.856741+0000","last_fullsized":"2024-04-24T16:00:54.856741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777254,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":254,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857313+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:00:54.857313+0000","last_peered":"2024-04-24T16:00:54.857313+0000","last_clean":"2024-04-24T16:00:54.857313+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:00:54.857313+0000","last_undegraded":"2024-04-24T16:00:54.857313+0000","last_fullsized":"2024-04-24T16:00:54.857313+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"86'91","reported_seq":497,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.833678+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:56.833678+0000","last_peered":"2024-04-24T16:00:56.833678+0000","last_clean":"2024-04-24T16:00:56.833678+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:56.833678+0000","last_undegraded":"2024-04-24T16:00:56.833678+0000","last_fullsized":"2024-04-24T16:00:56.833678+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":13111412,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":495,"num_read_kb":11079,"num_write":84,"num_write_kb":8504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857312+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:54.857312+0000","last_peered":"2024-04-24T16:00:54.857312+0000","last_clean":"2024-04-24T16:00:54.857312+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:54.857312+0000","last_undegraded":"2024-04-24T16:00:54.857312+0000","last_fullsized":"2024-04-24T16:00:54.857312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":141,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678380+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:55.678380+0000","last_peered":"2024-04-24T16:00:55.678380+0000","last_clean":"2024-04-24T16:00:55.678380+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:55.678380+0000","last_undegraded":"2024-04-24T16:00:55.678380+0000","last_fullsized":"2024-04-24T16:00:55.678380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"83'405","reported_seq":1542,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.818106+0000","last_change":"2024-04-24T16:00:25.382961+0000","last_active":"2024-04-24T16:00:56.818106+0000","last_peered":"2024-04-24T16:00:56.818106+0000","last_clean":"2024-04-24T16:00:56.818106+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:00:56.818106+0000","last_undegraded":"2024-04-24T16:00:56.818106+0000","last_fullsized":"2024-04-24T16:00:56.818106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:07.896694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":26224756,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":3434,"num_read_kb":48042,"num_write":396,"num_write_kb":41976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339960+0000","last_change":"2024-04-24T16:00:31.657960+0000","last_active":"2024-04-24T16:00:54.339960+0000","last_peered":"2024-04-24T16:00:54.339960+0000","last_clean":"2024-04-24T16:00:54.339960+0000","last_became_active":"2024-04-24T16:00:31.657630+0000","last_became_peered":"2024-04-24T16:00:31.657630+0000","last_unstale":"2024-04-24T16:00:54.339960+0000","last_undegraded":"2024-04-24T16:00:54.339960+0000","last_fullsized":"2024-04-24T16:00:54.339960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:57.306064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'6","reported_seq":99,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857342+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:54.857342+0000","last_peered":"2024-04-24T16:00:54.857342+0000","last_clean":"2024-04-24T16:00:54.857342+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:54.857342+0000","last_undegraded":"2024-04-24T16:00:54.857342+0000","last_fullsized":"2024-04-24T16:00:54.857342+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777332,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"83'227","reported_seq":1109,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.830472+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:56.830472+0000","last_peered":"2024-04-24T16:00:56.830472+0000","last_clean":"2024-04-24T16:00:56.830472+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:56.830472+0000","last_undegraded":"2024-04-24T16:00:56.830472+0000","last_fullsized":"2024-04-24T16:00:56.830472+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":31551699,"num_objects":23,"num_object_clones":9,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":814,"num_read_kb":63011,"num_write":204,"num_write_kb":56005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856845+0000","last_change":"2024-04-24T16:00:22.522734+0000","last_active":"2024-04-24T16:00:54.856845+0000","last_peered":"2024-04-24T16:00:54.856845+0000","last_clean":"2024-04-24T16:00:54.856845+0000","last_became_active":"2024-04-24T16:00:22.364893+0000","last_became_peered":"2024-04-24T16:00:22.364893+0000","last_unstale":"2024-04-24T16:00:54.856845+0000","last_undegraded":"2024-04-24T16:00:54.856845+0000","last_fullsized":"2024-04-24T16:00:54.856845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.344126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":85,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339909+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:54.339909+0000","last_peered":"2024-04-24T16:00:54.339909+0000","last_clean":"2024-04-24T16:00:54.339909+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:54.339909+0000","last_undegraded":"2024-04-24T16:00:54.339909+0000","last_fullsized":"2024-04-24T16:00:54.339909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"83'120","reported_seq":1140,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.810830+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:56.810830+0000","last_peered":"2024-04-24T16:00:56.810830+0000","last_clean":"2024-04-24T16:00:56.810830+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:56.810830+0000","last_undegraded":"2024-04-24T16:00:56.810830+0000","last_fullsized":"2024-04-24T16:00:56.810830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:29:42.919951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":26222777,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":4,"num_read":1157,"num_read_kb":47467,"num_write":88,"num_write_kb":33363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858826+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:54.858826+0000","last_peered":"2024-04-24T16:00:54.858826+0000","last_clean":"2024-04-24T16:00:54.858826+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:54.858826+0000","last_undegraded":"2024-04-24T16:00:54.858826+0000","last_fullsized":"2024-04-24T16:00:54.858826+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"83'7","reported_seq":120,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858989+0000","last_change":"2024-04-24T16:00:31.701555+0000","last_active":"2024-04-24T16:00:54.858989+0000","last_peered":"2024-04-24T16:00:54.858989+0000","last_clean":"2024-04-24T16:00:54.858989+0000","last_became_active":"2024-04-24T16:00:31.657902+0000","last_became_peered":"2024-04-24T16:00:31.657902+0000","last_unstale":"2024-04-24T16:00:54.858989+0000","last_undegraded":"2024-04-24T16:00:54.858989+0000","last_fullsized":"2024-04-24T16:00:54.858989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:57.117033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":7,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388684,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'9","reported_seq":144,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339990+0000","last_change":"2024-04-24T16:00:22.470388+0000","last_active":"2024-04-24T16:00:54.339990+0000","last_peered":"2024-04-24T16:00:54.339990+0000","last_clean":"2024-04-24T16:00:54.339990+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:00:54.339990+0000","last_undegraded":"2024-04-24T16:00:54.339990+0000","last_fullsized":"2024-04-24T16:00:54.339990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165955,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856704+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:54.856704+0000","last_peered":"2024-04-24T16:00:54.856704+0000","last_clean":"2024-04-24T16:00:54.856704+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:54.856704+0000","last_undegraded":"2024-04-24T16:00:54.856704+0000","last_fullsized":"2024-04-24T16:00:54.856704+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":97110644,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":5900,"num_read_kb":169599,"num_write":772,"num_write_kb":139848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181395456,"data_stored":194811112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218321,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":123,"num_read_kb":4808,"num_write":57,"num_write_kb":131224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":71303335,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268533760,"data_stored":268436642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393452,"kb_used_data":96096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93978388,"statfs":{"total":96636764160,"available":96233869312,"internally_reserved":0,"allocated":98402304,"data_stored":104534385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.34999999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.39500000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.315}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.88}]}]},{"osd":4,"up_from":11,"seq":47244640287,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385280,"kb_used_data":83824,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93986560,"statfs":{"total":96636764160,"available":96242237440,"internally_reserved":0,"allocated":85835776,"data_stored":85186363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.39500000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.308}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640288,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315316,"kb_used_data":17956,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94056524,"statfs":{"total":96636764160,"available":96313880576,"internally_reserved":0,"allocated":18386944,"data_stored":17785446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53300000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.312}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28199999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365284,"kb_used_data":63832,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94006556,"statfs":{"total":96636764160,"available":96262713344,"internally_reserved":0,"allocated":65363968,"data_stored":64729224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.498},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.54300000000000004}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.52600000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.624}]}]},{"osd":0,"up_from":11,"seq":47244640288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389376,"kb_used_data":87924,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93982464,"statfs":{"total":96636764160,"available":96238043136,"internally_reserved":0,"allocated":90034176,"data_stored":89380683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.314}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.373}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46800000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.44400000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.41699999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398328,"kb_used_data":96876,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93973512,"statfs":{"total":96636764160,"available":96228876288,"internally_reserved":0,"allocated":99201024,"data_stored":105335157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33100000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.245},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.312}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.50600000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55600000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.308},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37769216,"data_stored":37748983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":50356224,"data_stored":50331956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67109033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34459648,"data_stored":34431277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56238080,"data_stored":62974279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25976832,"data_stored":25966708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34459648,"data_stored":34431277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":30261248,"data_stored":37007571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:59.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:00:59.323 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:59.326+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:00:59.337 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:00:59.337 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature disable rbd/testrbdnbd37497 journaling 2024-04-24T16:00:59.384 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.385 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:00:59.405 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config image rm rbd/testrbdnbd37497 rbd_discard_granularity_bytes 2024-04-24T16:00:59.427 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:00:59.430+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:00:59.461 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:59.461 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:00:59.461 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:00:59.462 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:00:59.462 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:00:59.464 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:00:59.464 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:00:59.464 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:00:59.528 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:59.531+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:59.560 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.561 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:00:59.572 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T16:00:59.630 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:00:59.633+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:00:59.642 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=41236 2024-04-24T16:00:59.643 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 41236 2024-04-24T16:00:59.643 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 41236 -C rbd-nbd 2024-04-24T16:00:59.648 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:00:59.648 INFO:tasks.workunit.client.0.smithi122.stdout: 41236 ? 00:00:00 rbd-nbd 2024-04-24T16:00:59.648 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature enable rbd/testrbdnbd37497 journaling 2024-04-24T16:00:59.695 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.696 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:00:59.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:00:59.710 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T16:00:59.711 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:59.711 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:59.711 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:59.711 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-04-24T16:00:59.712 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:00:59.712 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T16:00:59.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:59.713 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T16:00:59.727 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-04-24T16:00:58.871304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231919403,"num_objects":123,"num_object_clones":19,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":14,"num_read":6177,"num_read_kb":174537,"num_write":994,"num_write_kb":273494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":97059600,"num_keys_recovered":26,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1019,"ondisk_log_size":1019,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2247036,"kb_used_data":446508,"kb_used_omap":9,"kb_used_meta":1800438,"kb_avail":563984004,"statfs":{"total":579820584960,"available":577519620096,"internally_reserved":0,"allocated":457224192,"data_stored":466951258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843648528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6768,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":863,"num_read_kb":1682,"num_write":31,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015501"},"pg_stats":[{"pgid":"1.5","version":"83'6","reported_seq":131,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678349+0000","last_change":"2024-04-24T16:00:22.519701+0000","last_active":"2024-04-24T16:00:55.678349+0000","last_peered":"2024-04-24T16:00:55.678349+0000","last_clean":"2024-04-24T16:00:55.678349+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:00:55.678349+0000","last_undegraded":"2024-04-24T16:00:55.678349+0000","last_fullsized":"2024-04-24T16:00:55.678349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000273377,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858785+0000","last_change":"2024-04-24T15:59:52.272040+0000","last_active":"2024-04-24T16:00:54.858785+0000","last_peered":"2024-04-24T16:00:54.858785+0000","last_clean":"2024-04-24T16:00:54.858785+0000","last_became_active":"2024-04-24T15:59:50.269292+0000","last_became_peered":"2024-04-24T15:59:50.269292+0000","last_unstale":"2024-04-24T16:00:54.858785+0000","last_undegraded":"2024-04-24T16:00:54.858785+0000","last_fullsized":"2024-04-24T16:00:54.858785+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:09.970345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059829000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'9","reported_seq":141,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856898+0000","last_change":"2024-04-24T16:00:31.523686+0000","last_active":"2024-04-24T16:00:54.856898+0000","last_peered":"2024-04-24T16:00:54.856898+0000","last_clean":"2024-04-24T16:00:54.856898+0000","last_became_active":"2024-04-24T16:00:31.414295+0000","last_became_peered":"2024-04-24T16:00:31.414295+0000","last_unstale":"2024-04-24T16:00:54.856898+0000","last_undegraded":"2024-04-24T16:00:54.856898+0000","last_fullsized":"2024-04-24T16:00:54.856898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:31.661486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258716,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165824,"num_keys_recovered":11,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678359+0000","last_change":"2024-04-24T15:59:52.386446+0000","last_active":"2024-04-24T16:00:55.678359+0000","last_peered":"2024-04-24T16:00:55.678359+0000","last_clean":"2024-04-24T16:00:55.678359+0000","last_became_active":"2024-04-24T15:59:50.311549+0000","last_became_peered":"2024-04-24T15:59:50.311549+0000","last_unstale":"2024-04-24T16:00:55.678359+0000","last_undegraded":"2024-04-24T16:00:55.678359+0000","last_fullsized":"2024-04-24T16:00:55.678359+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:47:51.328515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027237399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"63'3","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858939+0000","last_change":"2024-04-24T15:59:52.271825+0000","last_active":"2024-04-24T16:00:54.858939+0000","last_peered":"2024-04-24T16:00:54.858939+0000","last_clean":"2024-04-24T16:00:54.858939+0000","last_became_active":"2024-04-24T15:59:50.266052+0000","last_became_peered":"2024-04-24T15:59:50.266052+0000","last_unstale":"2024-04-24T16:00:54.858939+0000","last_undegraded":"2024-04-24T16:00:54.858939+0000","last_fullsized":"2024-04-24T16:00:54.858939+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:33:08.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043626,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"81'6","reported_seq":111,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856741+0000","last_change":"2024-04-24T15:59:48.255670+0000","last_active":"2024-04-24T16:00:54.856741+0000","last_peered":"2024-04-24T16:00:54.856741+0000","last_clean":"2024-04-24T16:00:54.856741+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:00:54.856741+0000","last_undegraded":"2024-04-24T16:00:54.856741+0000","last_fullsized":"2024-04-24T16:00:54.856741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014736700000000001,"stat_sum":{"num_bytes":16777254,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":254,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857313+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:00:54.857313+0000","last_peered":"2024-04-24T16:00:54.857313+0000","last_clean":"2024-04-24T16:00:54.857313+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:00:54.857313+0000","last_undegraded":"2024-04-24T16:00:54.857313+0000","last_fullsized":"2024-04-24T16:00:54.857313+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"86'91","reported_seq":497,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.833678+0000","last_change":"2024-04-24T15:59:48.254895+0000","last_active":"2024-04-24T16:00:56.833678+0000","last_peered":"2024-04-24T16:00:56.833678+0000","last_clean":"2024-04-24T16:00:56.833678+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:00:56.833678+0000","last_undegraded":"2024-04-24T16:00:56.833678+0000","last_fullsized":"2024-04-24T16:00:56.833678+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040979900000000002,"stat_sum":{"num_bytes":13111412,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":495,"num_read_kb":11079,"num_write":84,"num_write_kb":8504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857312+0000","last_change":"2024-04-24T15:59:58.835399+0000","last_active":"2024-04-24T16:00:54.857312+0000","last_peered":"2024-04-24T16:00:54.857312+0000","last_clean":"2024-04-24T16:00:54.857312+0000","last_became_active":"2024-04-24T15:59:58.834518+0000","last_became_peered":"2024-04-24T15:59:58.834518+0000","last_unstale":"2024-04-24T16:00:54.857312+0000","last_undegraded":"2024-04-24T16:00:54.857312+0000","last_fullsized":"2024-04-24T16:00:54.857312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:38.543027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045928700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"50'8","reported_seq":141,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:55.678380+0000","last_change":"2024-04-24T15:59:48.269115+0000","last_active":"2024-04-24T16:00:55.678380+0000","last_peered":"2024-04-24T16:00:55.678380+0000","last_clean":"2024-04-24T16:00:55.678380+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:00:55.678380+0000","last_undegraded":"2024-04-24T16:00:55.678380+0000","last_fullsized":"2024-04-24T16:00:55.678380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000266564,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"83'405","reported_seq":1542,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.818106+0000","last_change":"2024-04-24T16:00:25.382961+0000","last_active":"2024-04-24T16:00:56.818106+0000","last_peered":"2024-04-24T16:00:56.818106+0000","last_clean":"2024-04-24T16:00:56.818106+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:00:56.818106+0000","last_undegraded":"2024-04-24T16:00:56.818106+0000","last_fullsized":"2024-04-24T16:00:56.818106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:07.896694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001510125,"stat_sum":{"num_bytes":26224756,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":3434,"num_read_kb":48042,"num_write":396,"num_write_kb":41976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339960+0000","last_change":"2024-04-24T16:00:31.657960+0000","last_active":"2024-04-24T16:00:54.339960+0000","last_peered":"2024-04-24T16:00:54.339960+0000","last_clean":"2024-04-24T16:00:54.339960+0000","last_became_active":"2024-04-24T16:00:31.657630+0000","last_became_peered":"2024-04-24T16:00:31.657630+0000","last_unstale":"2024-04-24T16:00:54.339960+0000","last_undegraded":"2024-04-24T16:00:54.339960+0000","last_fullsized":"2024-04-24T16:00:54.339960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:57.306064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'6","reported_seq":99,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.857342+0000","last_change":"2024-04-24T15:59:48.261027+0000","last_active":"2024-04-24T16:00:54.857342+0000","last_peered":"2024-04-24T16:00:54.857342+0000","last_clean":"2024-04-24T16:00:54.857342+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:00:54.857342+0000","last_undegraded":"2024-04-24T16:00:54.857342+0000","last_fullsized":"2024-04-24T16:00:54.857342+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024274100000000001,"stat_sum":{"num_bytes":16777332,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"83'227","reported_seq":1109,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.830472+0000","last_change":"2024-04-24T15:59:48.256670+0000","last_active":"2024-04-24T16:00:56.830472+0000","last_peered":"2024-04-24T16:00:56.830472+0000","last_clean":"2024-04-24T16:00:56.830472+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:00:56.830472+0000","last_undegraded":"2024-04-24T16:00:56.830472+0000","last_fullsized":"2024-04-24T16:00:56.830472+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020238449999999998,"stat_sum":{"num_bytes":31551699,"num_objects":23,"num_object_clones":9,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":814,"num_read_kb":63011,"num_write":204,"num_write_kb":56005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"63'2","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856845+0000","last_change":"2024-04-24T16:00:22.522734+0000","last_active":"2024-04-24T16:00:54.856845+0000","last_peered":"2024-04-24T16:00:54.856845+0000","last_clean":"2024-04-24T16:00:54.856845+0000","last_became_active":"2024-04-24T16:00:22.364893+0000","last_became_peered":"2024-04-24T16:00:22.364893+0000","last_unstale":"2024-04-24T16:00:54.856845+0000","last_undegraded":"2024-04-24T16:00:54.856845+0000","last_fullsized":"2024-04-24T16:00:54.856845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.344126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083227599999999996,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":85,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339909+0000","last_change":"2024-04-24T15:59:48.260432+0000","last_active":"2024-04-24T16:00:54.339909+0000","last_peered":"2024-04-24T16:00:54.339909+0000","last_clean":"2024-04-24T16:00:54.339909+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:00:54.339909+0000","last_undegraded":"2024-04-24T16:00:54.339909+0000","last_fullsized":"2024-04-24T16:00:54.339909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000359506,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"83'120","reported_seq":1140,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T16:00:56.810830+0000","last_change":"2024-04-24T15:59:48.269828+0000","last_active":"2024-04-24T16:00:56.810830+0000","last_peered":"2024-04-24T16:00:56.810830+0000","last_clean":"2024-04-24T16:00:56.810830+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:00:56.810830+0000","last_undegraded":"2024-04-24T16:00:56.810830+0000","last_fullsized":"2024-04-24T16:00:56.810830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:29:42.919951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078891200000000001,"stat_sum":{"num_bytes":26222777,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":4,"num_read":1157,"num_read_kb":47467,"num_write":88,"num_write_kb":33363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"63'1","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858826+0000","last_change":"2024-04-24T15:59:52.272045+0000","last_active":"2024-04-24T16:00:54.858826+0000","last_peered":"2024-04-24T16:00:54.858826+0000","last_clean":"2024-04-24T16:00:54.858826+0000","last_became_active":"2024-04-24T15:59:50.311622+0000","last_became_peered":"2024-04-24T15:59:50.311622+0000","last_unstale":"2024-04-24T16:00:54.858826+0000","last_undegraded":"2024-04-24T16:00:54.858826+0000","last_fullsized":"2024-04-24T16:00:54.858826+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:29:45.310595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053132799999999997,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"83'7","reported_seq":120,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.858989+0000","last_change":"2024-04-24T16:00:31.701555+0000","last_active":"2024-04-24T16:00:54.858989+0000","last_peered":"2024-04-24T16:00:54.858989+0000","last_clean":"2024-04-24T16:00:54.858989+0000","last_became_active":"2024-04-24T16:00:31.657902+0000","last_became_peered":"2024-04-24T16:00:31.657902+0000","last_unstale":"2024-04-24T16:00:54.858989+0000","last_undegraded":"2024-04-24T16:00:54.858989+0000","last_fullsized":"2024-04-24T16:00:54.858989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:57.117033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237297,"stat_sum":{"num_bytes":8388800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":7,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388684,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'9","reported_seq":144,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.339990+0000","last_change":"2024-04-24T16:00:22.470388+0000","last_active":"2024-04-24T16:00:54.339990+0000","last_peered":"2024-04-24T16:00:54.339990+0000","last_clean":"2024-04-24T16:00:54.339990+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:00:54.339990+0000","last_undegraded":"2024-04-24T16:00:54.339990+0000","last_fullsized":"2024-04-24T16:00:54.339990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029551099999999998,"stat_sum":{"num_bytes":25165955,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"48'2","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-24T16:00:54.856704+0000","last_change":"2024-04-24T15:59:52.271708+0000","last_active":"2024-04-24T16:00:54.856704+0000","last_peered":"2024-04-24T16:00:54.856704+0000","last_clean":"2024-04-24T16:00:54.856704+0000","last_became_active":"2024-04-24T15:59:50.310968+0000","last_became_peered":"2024-04-24T15:59:50.310968+0000","last_unstale":"2024-04-24T16:00:54.856704+0000","last_undegraded":"2024-04-24T16:00:54.856704+0000","last_fullsized":"2024-04-24T16:00:54.856704+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.252354+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.252354+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:54.017210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037161099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":35,"num_keys_recovered":2,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":97110644,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":5900,"num_read_kb":169599,"num_write":772,"num_write_kb":139848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181395456,"data_stored":194811112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218321,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":123,"num_read_kb":4808,"num_write":57,"num_write_kb":131224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":71303335,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268533760,"data_stored":268436642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393452,"kb_used_data":96096,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93978388,"statfs":{"total":96636764160,"available":96233869312,"internally_reserved":0,"allocated":98402304,"data_stored":104534385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.34999999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.39500000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.315}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.88}]}]},{"osd":4,"up_from":11,"seq":47244640287,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385280,"kb_used_data":83824,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93986560,"statfs":{"total":96636764160,"available":96242237440,"internally_reserved":0,"allocated":85835776,"data_stored":85186363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.39500000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.308}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32700000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640288,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315316,"kb_used_data":17956,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94056524,"statfs":{"total":96636764160,"available":96313880576,"internally_reserved":0,"allocated":18386944,"data_stored":17785446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53300000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.312}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28199999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365284,"kb_used_data":63832,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94006556,"statfs":{"total":96636764160,"available":96262713344,"internally_reserved":0,"allocated":65363968,"data_stored":64729224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.498},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.54300000000000004}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.52600000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.624}]}]},{"osd":0,"up_from":11,"seq":47244640288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389376,"kb_used_data":87924,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93982464,"statfs":{"total":96636764160,"available":96238043136,"internally_reserved":0,"allocated":90034176,"data_stored":89380683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.314}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.373}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46800000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.44400000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.41699999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398328,"kb_used_data":96876,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93973512,"statfs":{"total":96636764160,"available":96228876288,"internally_reserved":0,"allocated":99201024,"data_stored":105335157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33100000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.245},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.312}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.50600000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55600000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.308},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37769216,"data_stored":37748983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":50356224,"data_stored":50331956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":67109033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34459648,"data_stored":34431277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56238080,"data_stored":62974279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25976832,"data_stored":25966708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34459648,"data_stored":34431277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":30261248,"data_stored":37007571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:00:59.728 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T16:00:59.728 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:00:59.729 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T16:00:59.729 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:00:59.729 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:00:59.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T16:00:59.731 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:00:59.734+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:00:59.735 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature disable rbd/testrbdnbd37497 journaling 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 41236 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=41236 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:00:59.803 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:00:59.804 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:00:59.804 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:00:59.806 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:00:59.806 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:00:59.806 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:00:59.834 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:00:59.837+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:00:59.852 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:00:59.852 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:00:59.854 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:00:59.855 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T16:00:59.855 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:00:59.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:00:59.880 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-04-24T16:00:59.936 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:00:59.938+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:00.005 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:00.005 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:00.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:01:00.037 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:00.040+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:00.138 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:00.141+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:00.154 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:00.155 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:00.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:01:00.240 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:00.243+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:00.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:01:00.341 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:00.343+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:00.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:01:00.398 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:01:00.398 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^41236\b' 2024-04-24T16:01:00.398 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^41236\b' 2024-04-24T16:01:00.442 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:00.445+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:00.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:00.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:01:00.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *41236\b' 2024-04-24T16:01:00.450 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *41236\b' 2024-04-24T16:01:00.454 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:00.454 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:01:00.487 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:01:00.506 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:01:00.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:01:00.543 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:00.546+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:00.547 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:00.547 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:00.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:01:00.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:01:00.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:01:00.592 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:01:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:01:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:01:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:01:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:01:00.594 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:01:00.594 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:01:00.648 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:00.651+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:00.710 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:00.710 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:00.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:00.710 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.405208243000004, 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.395588179000001, 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.715 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.375187612000005, 2024-04-24T16:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:00.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:01:00.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.362255809999994, 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:01:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:00.729 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.362222059000004, 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:00.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:01:00.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.734 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.360699648999997, 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:01:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.739 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.355288533000007, 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:01:00.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:01:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.347540015000007, 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:01:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:01:00.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:01:00.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.345409992, 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.329707034999998, 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:00.759 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.329679048000003, 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.312907972000005, 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.764 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:00.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:01:00.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.769 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.307564447999994, 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.295354203000002, 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.287919037999998, 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.782 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:01:00.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.103663139000005, 2024-04-24T16:01:00.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:00.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:00.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:01:00.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:01:00.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.796 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.099427211999995, 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 72.092587445999996, 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:00.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.338531027999998, 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:00.812 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.336306364000002, 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:01:00.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:01:00.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:01:00.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:01:00.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:00.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:00.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:00.822 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:00.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:01:00.825 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:00.826+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:00.924 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:00.927+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:00.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:01:00.956 INFO:tasks.workunit.client.0.smithi122.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-04-24T16:01:00.971 INFO:tasks.workunit.client.0.smithi122.stderr:+ EXPECTED=2 2024-04-24T16:01:00.972 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:01:00.972 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:01:00.972 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:01:00.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:01:00.973 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:01:00.975 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:01:00.975 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:01:00.975 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd37497 2024-04-24T16:01:01.010 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:01.010 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:01.010 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:01.010 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:01.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:01:01.025 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:01.027+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:01.084 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-04-24T16:01:01.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:01:01.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:01:01.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:01:01.085 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:01:01.085 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T16:01:01.125 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:01.128+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:01.160 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=41358 2024-04-24T16:01:01.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 41358 2024-04-24T16:01:01.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 41358 -C rbd-nbd 2024-04-24T16:01:01.163 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:01.164 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:01.166 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:01:01.166 INFO:tasks.workunit.client.0.smithi122.stdout: 41358 ? 00:00:00 rbd-nbd 2024-04-24T16:01:01.166 INFO:tasks.workunit.client.0.smithi122.stderr:++ ps -p 41358 -T 2024-04-24T16:01:01.166 INFO:tasks.workunit.client.0.smithi122.stderr:++ grep -c io_context_pool 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ ACTUAL=2 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 2 -eq 2 ']' 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 41358 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=41358 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:01:01.173 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:01:01.174 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:01:01.174 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:01:01.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:01:01.176 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:01:01.176 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:01:01.176 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:01:01.226 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:01.228+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:01.327 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:01.329+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:01.359 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.981665493000001, 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020635100000000002, 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379139+0000", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379137+0000", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379946+0000", 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080930100000000001 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379971+0000", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.379983+0000", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1805e-05 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380070+0000", 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6971999999999993e-05 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380081+0000", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1080999999999999e-05 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381202+0000", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001120823 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5404.0:13 3.0 e66/65 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 66'256, mlcod=0'0)", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:01:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.980802146000002, 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021205970000000001, 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380002+0000", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380001+0000", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380011+0000", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8330000000000006e-06 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380016+0000", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000004e-06 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.380019+0000", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.264e-06 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381229+0000", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012094479999999999 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.381254+0000", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5829e-05 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382090+0000", 2024-04-24T16:01:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083569700000000005 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.382123+0000", 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2366000000000001e-05 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head, version: 56'176, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head@56'176, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.977419881000003, 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001630619, 2024-04-24T16:01:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383384+0000", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383383+0000", 2024-04-24T16:01:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383941+0000", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055831999999999995 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383968+0000", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6182000000000001e-05 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.383981+0000", 2024-04-24T16:01:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3193999999999999e-05 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384078+0000", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7522000000000006e-05 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.384090+0000", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1589e-05 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.385015+0000", 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092503300000000002 2024-04-24T16:01:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.927224543000001, 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:01:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:01:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.922191114, 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001650199, 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438613+0000", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.438612+0000", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439170+0000", 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055779899999999999 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439189+0000", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9616e-05 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439198+0000", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1079999999999999e-06 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439286+0000", 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7225999999999995e-05 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.439299+0000", 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3159999999999999e-05 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.440263+0000", 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096450699999999997 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head, version: 57'221, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head@57'221, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.918236366000002, 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018634210000000001, 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442568+0000", 2024-04-24T16:01:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.442567+0000", 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443139+0000", 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057271799999999995 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443164+0000", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4841000000000002e-05 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443177+0000", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2480000000000001e-05 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443244+0000", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7921000000000002e-05 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.443255+0000", 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0161e-05 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.444431+0000", 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001176519 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head, version: 59'251, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3000966f:ns::rbd_data.1308dd266c0.0000000000000001:head@59'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:01:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.893306199000001, 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001865773, 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467498+0000", 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.467497+0000", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468235+0000", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073881800000000003 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468260+0000", 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4250999999999999e-05 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468275+0000", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5543e-05 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468350+0000", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4506999999999997e-05 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.468357+0000", 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1280000000000002e-06 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.469364+0000", 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010068589999999999 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 35.978181306000003, 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:76 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'296, mlcod=80'294)", 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.029588173, 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055479819999999999, 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331215+0000", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331221+0000", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331233+0000", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1596999999999999e-05 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331240+0000", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8920000000000003e-06 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331282+0000", 2024-04-24T16:01:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1681999999999998e-05 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331301+0000", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9613999999999999e-05 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336737+0000", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054352849999999998 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336764+0000", 2024-04-24T16:01:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7348e-05 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:77 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'297, mlcod=80'294)", 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.028686227000001, 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046695269999999997, 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332117+0000", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332125+0000", 2024-04-24T16:01:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332139+0000", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3392e-05 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332146+0000", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3050000000000004e-06 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332193+0000", 2024-04-24T16:01:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6969999999999999e-05 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332216+0000", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3156e-05 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336766+0000", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045501680000000003 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336787+0000", 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1287000000000001e-05 2024-04-24T16:01:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "pg_update_log_missing(3.0 epoch 80/67 rep_tid 482 entries 80'298 (0'0) error 3:372fdc5e:::rbd_header.129ad8b6ae3f:head by client.5285.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 80'294)", 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.027890197000001, 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038821250000000002, 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332913+0000", 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332923+0000", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9159999999999996e-06 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332940+0000", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6986e-05 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332949+0000", 2024-04-24T16:01:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8289999999999997e-06 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333006+0000", 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6969999999999998e-05 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336796+0000", 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003790093 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:79 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'299, mlcod=80'294)", 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.027366178000001, 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003383718, 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333437+0000", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333444+0000", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4290000000000004e-06 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333455+0000", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0828e-05 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333461+0000", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4639999999999997e-06 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333503+0000", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2086999999999999e-05 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333523+0000", 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0094000000000001e-05 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336797+0000", 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032740569999999999 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336822+0000", 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4691000000000001e-05 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:82 3.0 e80/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 80'301, mlcod=80'294)", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.026559935000002, 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003048916, 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334243+0000", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334250+0000", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1530000000000002e-06 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334260+0000", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8810000000000008e-06 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334264+0000", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4410000000000003e-06 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334305+0000", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0880999999999997e-05 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334336+0000", 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1554999999999999e-05 2024-04-24T16:01:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337270+0000", 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002933546 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337293+0000", 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.315e-05 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:84 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'303, mlcod=80'299)", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.020546915000001, 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044460719999999997, 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340256+0000", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340263+0000", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340274+0000", 2024-04-24T16:01:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0509000000000001e-05 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340281+0000", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7290000000000003e-06 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340339+0000", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8041000000000002e-05 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340361+0000", 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1892e-05 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344658+0000", 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042975230000000001 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344703+0000", 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:85 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'305, mlcod=80'299)", 2024-04-24T16:01:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.019770914999999, 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042239729999999998, 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341038+0000", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9869999999999996e-06 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341054+0000", 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5784e-05 2024-04-24T16:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341071+0000", 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341122+0000", 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.109e-05 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341142+0000", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9956999999999999e-05 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345218+0000", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040761019999999999 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345257+0000", 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8852e-05 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:101 3.0 e80/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 80'318, mlcod=80'314)", 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.004303656000001, 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041012970000000003, 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356507+0000", 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0140000000000001e-06 2024-04-24T16:01:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356517+0000", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.005e-05 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356523+0000", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4489999999999996e-06 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356566+0000", 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2168e-05 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356582+0000", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6711e-05 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360572+0000", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039900580000000003 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360602+0000", 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9408e-05 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:102 3.0 e80/67 3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 80'320, mlcod=80'314)", 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.998616715000001, 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044501560000000003, 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362193+0000", 2024-04-24T16:01:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0310000000000004e-06 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362203+0000", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0345e-05 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362208+0000", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9300000000000002e-06 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362246+0000", 2024-04-24T16:01:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8188000000000001e-05 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362259+0000", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3057999999999999e-05 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366606+0000", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043471830000000001 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366638+0000", 2024-04-24T16:01:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1063999999999999e-05 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:107 3.0 e80/67 3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 80'325, mlcod=80'320)", 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.990754035999998, 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045367460000000004, 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370049+0000", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370055+0000", 2024-04-24T16:01:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2560000000000002e-06 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370064+0000", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1910000000000006e-06 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370068+0000", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.267e-06 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370105+0000", 2024-04-24T16:01:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6418999999999998e-05 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370130+0000", 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5082000000000001e-05 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374558+0000", 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004428871 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374587+0000", 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.832e-05 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:139 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'343, mlcod=80'339)", 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.961779726, 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001640063, 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399024+0000", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399023+0000", 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399118+0000", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4485000000000001e-05 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399139+0000", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0815e-05 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399145+0000", 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4969999999999998e-06 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399181+0000", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5942000000000002e-05 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399204+0000", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2826000000000001e-05 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400627+0000", 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014233449999999999 2024-04-24T16:01:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400664+0000", 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7036999999999998e-05 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:140 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'344, mlcod=80'339)", 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 16.960956645, 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021562809999999999, 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399847+0000", 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399951+0000", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000104269 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399966+0000", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5350999999999999e-05 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.399973+0000", 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2750000000000001e-06 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400011+0000", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8074000000000002e-05 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.400032+0000", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0789e-05 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.401963+0000", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019310410000000001 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.402004+0000", 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1174999999999998e-05 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:01.438 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:01.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:01:01.440 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:01.442+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:01.541 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:01.544+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:01.575 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:01.575 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:01.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:01:01.641 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:01.644+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:01.707 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:01.707 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:01.707 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:01.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:01:01.760 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:01.763+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:01.771 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:01:01.771 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^41358\b' 2024-04-24T16:01:01.771 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^41358\b' 2024-04-24T16:01:01.823 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:01.824 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:01:01.824 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *41358\b' 2024-04-24T16:01:01.824 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *41358\b' 2024-04-24T16:01:01.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:01.829 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:01:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:01.841 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:01.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:01.841 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.531853544000001, 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.523887513999995, 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:01:01.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:01:01.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:01:01.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:01:01.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.523853318999997, 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:01:01.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:01:01.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.523834680999997, 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.856 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:01:01.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.516511944000001, 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:01:01.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.866 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.494852921000003, 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:01:01.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:01:01.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.490604286999996, 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:01:01.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.490569769000004, 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:01:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:01:01.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:01:01.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:01:01.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:01:01.885 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:01:01.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.460408665000003, 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:01.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.409313725999993, 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:01.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:01:01.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:01:01.896 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.355795213999997, 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.232455053999999, 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.902 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.215585598999994, 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:01:01.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.207369173999993, 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:01:01.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:01:01.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:01:01.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.193829026000003, 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.191944356999997, 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.920 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:01.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:01:01.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:01:01.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:01:01.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.171336097999998, 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.925 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:01:01.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.161414597999993, 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.930 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.156766340000004, 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:01:01.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:01:01.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.155272549000003, 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:01.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:01:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:01:01.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:01:01.941 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.942 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.942 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.942 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:01.942 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:01.942 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:01.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T16:01:01.944 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:01.946+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:01.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:01:02.044 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:02.047+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:02.110 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:02.110 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:02.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:01:02.145 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:02.148+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:02.246 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:02.249+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:02.283 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:02.283 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:02.283 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:02.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:01:02.330 INFO:tasks.workunit.client.0.smithi122.stderr:+ EXPECTED=7 2024-04-24T16:01:02.331 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd37497 2024-04-24T16:01:02.331 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-04-24T16:01:02.331 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-04-24T16:01:02.332 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T16:01:02.332 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-04-24T16:01:02.334 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-04-24T16:01:02.334 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-04-24T16:01:02.334 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd37497 2024-04-24T16:01:02.347 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:02.350+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:02.448 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:02.451+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:02.452 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-04-24T16:01:02.452 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-04-24T16:01:02.453 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-04-24T16:01:02.453 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-04-24T16:01:02.453 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T16:01:02.453 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd37497'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T16:01:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:02.460 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:02.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:02.460 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.155708363000002, 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:01:02.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:01:02.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:01:02.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.154561576999996, 2024-04-24T16:01:02.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:01:02.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.138709454999997, 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:01:02.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:01:02.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.115944823999996, 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:01:02.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.105826468000004, 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:01:02.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:01:02.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:01:02.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.077883353000004, 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:01:02.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:02.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.077865527, 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:02.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:01:02.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:01:02.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:01:02.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.068248128999997, 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.493 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:01:02.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:01:02.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.064519218000001, 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.497 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 74.038635154999994, 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.505 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.982413515000005, 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:01:02.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:01:02.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.982392404999999, 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.869517560000006, 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.516 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:01:02.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:01:02.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.860099790000007, 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.520 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:01:02.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.843583624000004, 2024-04-24T16:01:02.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.843027207999995, 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:01:02.528 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:02.533 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.793796803999996, 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:01:02.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:01:02.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.782118976999996, 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.538 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:01:02.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:01:02.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.780771217999998, 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:01:02.542 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:01:02.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:01:02.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:01:02.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 73.772643638000005, 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:01:02.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:02.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:01:02.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:02.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:02.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:02.553 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=41445 2024-04-24T16:01:02.553 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 41445 2024-04-24T16:01:02.553 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 41445 -C rbd-nbd 2024-04-24T16:01:02.553 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-04-24T16:01:02.554 INFO:tasks.workunit.client.0.smithi122.stdout: 41445 ? 00:00:00 rbd-nbd 2024-04-24T16:01:02.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:++ ps -p 41445 -T 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:++ grep -c io_context_pool 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ ACTUAL=7 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 7 -eq 7 ']' 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 41445 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=41445 2024-04-24T16:01:02.556 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-04-24T16:01:02.557 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T16:01:02.557 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:02.556+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:02.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:01:02.654 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:02.657+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:02.719 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:02.720 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:02.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:01:02.755 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:02.758+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:02.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:01:02.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:01:02.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:01:02.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:01:02.849 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:01:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:02.853 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:02.853 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:02.853 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:02.855 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:02.858+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:02.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:01:02.868 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:01:02.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:01:02.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:01:02.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:01:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:01:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:01:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:01:02.888 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-04-24T16:01:02.957 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:02.960+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:01:03.058 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:03.061+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:03.138 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-04-24T16:01:03.138 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^41445\b' 2024-04-24T16:01:03.138 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^41445\b' 2024-04-24T16:01:03.160 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:03.162+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.188 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:03.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-04-24T16:01:03.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *41445\b' 2024-04-24T16:01:03.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *41445\b' 2024-04-24T16:01:03.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-04-24T16:01:03.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:01:03.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:01:03.261 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:03.264+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:03.335 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.336 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:01:03.392 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:03.395+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:03.462 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:03.465+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:03.488 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.488 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:01:03.584 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:03.587+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:03.664 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:03.667+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:03.682 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.682 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:01:03.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-04-24T16:01:03.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ echo OK 2024-04-24T16:01:03.695 INFO:tasks.workunit.client.0.smithi122.stdout:OK 2024-04-24T16:01:03.696 INFO:tasks.workunit.client.0.smithi122.stderr:+ cleanup 2024-04-24T16:01:03.696 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns s 2024-04-24T16:01:03.696 INFO:tasks.workunit.client.0.smithi122.stderr:+ set +e 2024-04-24T16:01:03.696 INFO:tasks.workunit.client.0.smithi122.stderr:+ mount 2024-04-24T16:01:03.696 INFO:tasks.workunit.client.0.smithi122.stderr:+ fgrep /tmp/tmp.P3d5b6KQ4E/mnt 2024-04-24T16:01:03.700 INFO:tasks.workunit.client.0.smithi122.stderr:+ rm -Rf /tmp/tmp.P3d5b6KQ4E 2024-04-24T16:01:03.707 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n '' ']' 2024-04-24T16:01:03.708 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-04-24T16:01:03.708 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' status testrbdnbd37497 2024-04-24T16:01:03.748 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-04-24T16:01:03.751 INFO:tasks.workunit.client.0.smithi122.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-04-24T16:01:03.751 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:01:03.765 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:03.768+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:03.866 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:03.869+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:03.876 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:03.876 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:03.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:01:03.968 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:03.970+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:04.028 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.028 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:01:04.069 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:04.071+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:04.170 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:04.173+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:04.178 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.178 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:01:04.252 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' status testrbdnbd37497 2024-04-24T16:01:04.252 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'Watchers: none' 2024-04-24T16:01:04.271 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:04.273+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:04.298 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-04-24T16:01:04.299 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-04-24T16:01:04.299 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd37497 2024-04-24T16:01:04.371 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:04.374+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:04.380 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.380 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:01:04.472 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:04.475+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:04.530 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.530 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:01:04.587 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:04.590+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:04.675 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:04.677+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:01:04.776 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:04.779+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:04.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:01:04.865 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T16:01:04.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T16:01:04.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:01:04.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:01:04.912 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:04.914+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:04.978 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:04.981+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:05.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:01:05.079 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:05.082+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:05.082 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:01:05.101 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:01:05.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:01:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:05.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:01:05.142 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:01:05.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:01:05.180 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:05.183+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:05.181 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:01:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:01:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:01:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:01:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:01:05.183 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:01:05.183 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:01:05.281 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:05.284+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:05.298 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:05.299 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:05.299 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:05.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:01:05.382 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:05.385+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.128030960999993, 2024-04-24T16:01:05.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:05.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:01:05.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:01:05.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.126884175000001, 2024-04-24T16:01:05.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:01:05.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:01:05.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:01:05.448 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.111032053000002, 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.449 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:01:05.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:01:05.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:01:05.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:01:05.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.088267422000001, 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.456 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:01:05.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:01:05.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:01:05.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:01:05.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.078149065999995, 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.466 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:01:05.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.050205950999995, 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:01:05.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:01:05.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.050188125000005, 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:01:05.482 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:05.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:01:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.040570727000002, 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:01:05.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:01:05.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.036841816000006, 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:01:05.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:01:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.010957753, 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:01:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.954736112999996, 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:01:05.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:01:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:01:05.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:01:05.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.954715003000004, 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:01:05.508 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:05.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:01:05.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.841840157999997, 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.832422387999998, 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:05.517 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:01:05.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:01:05.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.815906221999995, 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.521 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.815349806, 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.766119402000001, 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:01:05.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.533 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.754441575000001, 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:05.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:01:05.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.537 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.753093816000003, 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:01:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.744966235999996, 2024-04-24T16:01:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:05.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:01:05.549 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:05.551+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:05.649 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:05.651+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:05.672 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:05.672 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:05.673 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:05.673 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:05.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:01:05.768 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:05.770+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:05.848 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:05.848 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:05.849 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:05.849 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:05.849 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:05.851 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:05.854+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:05.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:01:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T16:01:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:01:05.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T16:01:05.953 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:05.955+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:06.027 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.704524015999993, 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:01:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:01:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:01:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.702719435999995, 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:01:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:01:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.702688068, 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.678964342, 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:01:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.651106604000006, 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:01:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:01:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:01:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:01:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:01:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.646477098999995, 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:01:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:01:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.639669556000001, 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.609129328999998, 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.593489649000006, 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:01:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.554314399999996, 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:01:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.541696279999996, 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.419300143000001, 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:01:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:01:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:01:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.405770935000007, 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.395036606999994, 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:01:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.380740463999999, 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.348513371999999, 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:01:06.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:01:06.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.342417530999995, 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:06.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.338962973999998, 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 77.303518853, 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 70.252059087999996, 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:06.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:06.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:01:06.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:01:06.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:06.125 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:06.126 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:06.129+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:06.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:01:06.228 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:06.230+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:06.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:01:06.329 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:06.331+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:06.418 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:06.419 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:06.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:01:06.431 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:06.433+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:06.531 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:06.534+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:06.571 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:06.571 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:35 1.2 e44/12 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.254033114999999, 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016826137000000001, 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318760+0000", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.318759+0000", 2024-04-24T16:01:06.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325776+0000", 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007017789 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325802+0000", 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5737000000000001e-05 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325814+0000", 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1999e-05 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325863+0000", 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9137000000000001e-05 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.325898+0000", 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5120999999999998e-05 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335545+0000", 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096462179999999998 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335586+0000", 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1597999999999997e-05 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:37 1.6 e44/12 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'2, mlcod=0'0)", 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.252881436999999, 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015343307, 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319912+0000", 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.319911+0000", 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:06.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.326997+0000", 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0070855450000000004 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327012+0000", 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5780000000000001e-05 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327018+0000", 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0599999999999998e-06 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327085+0000", 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7265000000000001e-05 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.327103+0000", 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7773000000000001e-05 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335190+0000", 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080876109999999998 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335255+0000", 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4955000000000004e-05 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:40 1.0 e44/12 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'2, mlcod=0'0)", 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.237861652999996, 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010234521999999999, 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334932+0000", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.334930+0000", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339413+0000", 2024-04-24T16:01:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044829900000000001 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339431+0000", 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8104000000000001e-05 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339441+0000", 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7170000000000003e-06 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339482+0000", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1171999999999998e-05 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.339503+0000", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345077+0000", 2024-04-24T16:01:06.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055739780000000003 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345166+0000", 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8682000000000002e-05 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:42 1.0 e44/12 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'3, mlcod=44'1)", 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:01:06.586 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.216929315000002, 2024-04-24T16:01:06.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025336394000000002, 2024-04-24T16:01:06.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.587 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.589 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355864+0000", 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355863+0000", 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372521+0000", 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016658586 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372544+0000", 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3045e-05 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372554+0000", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372604+0000", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0284999999999997e-05 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.372625+0000", 2024-04-24T16:01:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381119+0000", 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084935329999999993 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381200+0000", 2024-04-24T16:01:06.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1598e-05 2024-04-24T16:01:06.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:4 3.3 e44/15 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'4, mlcod=42'2)", 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.216307361000005, 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026515970999999999, 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356486+0000", 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356485+0000", 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373482+0000", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016997826000000001 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373503+0000", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0148000000000001e-05 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373515+0000", 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.26e-05 2024-04-24T16:01:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373652+0000", 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013641 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.373681+0000", 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9886000000000002e-05 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.382899+0000", 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092175399999999998 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.383002+0000", 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010285000000000001 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:49 1.2 e44/12 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'2, mlcod=0'0)", 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.188322839999998, 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01169561, 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384470+0000", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.384468+0000", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389956+0000", 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054876489999999998 2024-04-24T16:01:06.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389977+0000", 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1128000000000002e-05 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.389989+0000", 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1276e-05 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390031+0000", 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2484000000000002e-05 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.390054+0000", 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3127999999999999e-05 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396107+0000", 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006052885 2024-04-24T16:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.396166+0000", 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9009999999999999e-05 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:50 1.2 e44/12 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'3, mlcod=0'0)", 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.175007149999999, 2024-04-24T16:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010867069, 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397786+0000", 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397785+0000", 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403515+0000", 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057301640000000003 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403539+0000", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4646000000000001e-05 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403550+0000", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0964e-05 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403599+0000", 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8461000000000002e-05 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403620+0000", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0784e-05 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408602+0000", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049828149999999998 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.408653+0000", 2024-04-24T16:01:06.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0769000000000001e-05 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:7 3.3 e44/15 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'5, mlcod=44'3)", 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.162295349000004, 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011588516, 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410498+0000", 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.410497+0000", 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416676+0000", 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061792069999999999 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416695+0000", 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9432e-05 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416703+0000", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416750+0000", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7392000000000003e-05 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416770+0000", 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.012e-05 2024-04-24T16:01:06.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.421991+0000", 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052203919999999999 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422086+0000", 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5654000000000003e-05 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:63 3.1 3:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.103706443999997, 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038865855999999997, 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:06.614 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469087+0000", 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.469085+0000", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478468+0000", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093821570000000003 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478494+0000", 2024-04-24T16:01:06.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6902000000000001e-05 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478504+0000", 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3640000000000005e-06 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.478559+0000", 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5717000000000002e-05 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507949+0000", 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.029389314999999999 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507953+0000", 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8820000000000003e-06 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:59 1.2 e44/12 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.084270988, 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010722282, 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488522+0000", 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.488520+0000", 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493885+0000", 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053644529999999999 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493911+0000", 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6712000000000001e-05 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493927+0000", 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5379999999999998e-05 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493975+0000", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229000000000002e-05 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.493999+0000", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3685000000000001e-05 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499182+0000", 2024-04-24T16:01:06.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051827280000000002 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.499245+0000", 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3059999999999996e-05 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4762.0:62 1.2 e44/12 1:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:01:06.623 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.071791005999998, 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012919013999999999, 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501002+0000", 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.501001+0000", 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507587+0000", 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065858399999999999 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507611+0000", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4411000000000001e-05 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507621+0000", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2539999999999998e-06 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507681+0000", 2024-04-24T16:01:06.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0715999999999999e-05 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.507714+0000", 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2391000000000002e-05 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513861+0000", 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061471690000000001 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.513921+0000", 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0492999999999998e-05 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:43 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.944355958000003, 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012221595, 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 43 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628437+0000", 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628435+0000", 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628448+0000", 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2525e-05 2024-04-24T16:01:06.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628470+0000", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2144999999999999e-05 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628481+0000", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0667e-05 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628543+0000", 2024-04-24T16:01:06.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2230000000000006e-05 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628689+0000", 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014594099999999999 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629151+0000", 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000462399 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630500+0000", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013488630000000001 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640540+0000", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010040494000000001 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640609+0000", 2024-04-24T16:01:06.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8418999999999997e-05 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640659+0000", 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0067e-05 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:44 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.635 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.943669211, 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011695646000000001, 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 44 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629124+0000", 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629123+0000", 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629129+0000", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.203e-06 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629138+0000", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629142+0000", 2024-04-24T16:01:06.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0999999999999997e-06 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629536+0000", 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039382399999999998 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.629662+0000", 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125227 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630092+0000", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043091400000000001 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630945+0000", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085265899999999997 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640714+0000", 2024-04-24T16:01:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097688359999999995 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640776+0000", 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2317000000000007e-05 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640820+0000", 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3435e-05 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:36 1.2 e44/12 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head v 44'6, mlcod=44'4)", 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.942514736999996, 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018018149000000001, 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630279+0000", 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.630278+0000", 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640412+0000", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010134248 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640424+0000", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1939e-05 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640432+0000", 2024-04-24T16:01:06.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1759999999999993e-06 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640481+0000", 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9175000000000003e-05 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640508+0000", 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6344999999999999e-05 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648268+0000", 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077601079999999999 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.648297+0000", 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9064000000000002e-05 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:38 1.0 e44/12 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.938867311999999, 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013993004, 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633926+0000", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.633925+0000", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639949+0000", 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060244419999999996 2024-04-24T16:01:06.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639975+0000", 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6029000000000001e-05 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.639992+0000", 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6469000000000001e-05 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640078+0000", 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5996999999999997e-05 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.640117+0000", 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9532e-05 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647842+0000", 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077245789999999996 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.647919+0000", 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7075000000000003e-05 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:47 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.918907220999998, 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010827064000000001, 2024-04-24T16:01:06.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 47 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653886+0000", 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653884+0000", 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653896+0000", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1593000000000001e-05 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653914+0000", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7802e-05 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653923+0000", 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7580000000000004e-06 2024-04-24T16:01:06.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653991+0000", 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7824999999999995e-05 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654125+0000", 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337599999999999 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:06.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.654476+0000", 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035125 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655233+0000", 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075683 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664646+0000", 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094129179999999993 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664683+0000", 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7107000000000002e-05 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.664713+0000", 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0337000000000001e-05 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.657 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.4872.0:46 1.0 e44/12 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.917499312000004, 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085380790000000005, 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655294+0000", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655293+0000", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659146+0000", 2024-04-24T16:01:06.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038534960000000001 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659159+0000", 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2337999999999999e-05 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659166+0000", 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0119999999999999e-06 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659204+0000", 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8526999999999999e-05 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.659223+0000", 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9046999999999999e-05 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663787+0000", 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045631589999999998 2024-04-24T16:01:06.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.663832+0000", 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5485999999999998e-05 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:48 3.1 3:a42a88df:ns::rbd_object_map.1308dd266c0:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:01:06.662 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.916863262999996, 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097638480000000003, 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:01:06.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655930+0000", 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655929+0000", 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655935+0000", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.241e-06 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655948+0000", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2377e-05 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655955+0000", 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6499999999999996e-06 2024-04-24T16:01:06.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655990+0000", 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5095000000000002e-05 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656065+0000", 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-05 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:06.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.656295+0000", 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000229853 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.657018+0000", 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000722641 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665609+0000", 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085909000000000003 2024-04-24T16:01:06.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665667+0000", 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8708000000000001e-05 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.665694+0000", 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6539000000000001e-05 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:23 3.3 e48/15 3:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head v 48'13, mlcod=44'11)", 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 73.733007752999995, 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012674942999999999, 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839786+0000", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.839784+0000", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846580+0000", 2024-04-24T16:01:06.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067954139999999996 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846606+0000", 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5846e-05 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846616+0000", 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774999999999999e-05 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846666+0000", 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0141e-05 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.846687+0000", 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0162000000000001e-05 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852419+0000", 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057322019999999996 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:52.852460+0000", 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1666999999999999e-05 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.4.0:102 3.3 e59/15 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 59'91, mlcod=59'89)", 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.094264166000002, 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011123375, 2024-04-24T16:01:06.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478529+0000", 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:06.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.478528+0000", 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482917+0000", 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043897449999999996 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:06.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482944+0000", 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6202000000000001e-05 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.482956+0000", 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2311e-05 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483020+0000", 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3670999999999994e-05 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.483048+0000", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8722e-05 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489582+0000", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065336150000000004 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:11.489652+0000", 2024-04-24T16:01:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0568999999999995e-05 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:06.678 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:06.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:01:06.680 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:06.682+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:06.780 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:06.783+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:06.824 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:06.824 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:06.824 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:06.824 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:06.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:01:06.881 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:06.884+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:06.914 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T16:01:06.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T16:01:06.982 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:06.984+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:06.987 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:06.988 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:06.988 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:06.988 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:06.988 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:07.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:01:07.083 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:07.085+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:07.184 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:07.187+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:07.193 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.887573400999997, 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:07.198 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.877953336999994, 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:07.202 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.857552769999998, 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:01:07.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:01:07.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.844620968000001, 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:01:07.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:01:07.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.844587216999997, 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:01:07.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:01:07.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:07.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:01:07.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:01:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.843064807000005, 2024-04-24T16:01:07.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:01:07.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:01:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.837653691, 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.239 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:01:07.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:01:07.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.829905173, 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.243 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:01:07.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:01:07.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.827775149999994, 2024-04-24T16:01:07.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.250 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:07.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:01:07.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:01:07.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.812072193000006, 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.259 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.812044205999996, 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:01:07.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:01:07.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.795273129999998, 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:01:07.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:01:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:01:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.789929606000001, 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.777719360999996, 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:01:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:07.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.770284196000006, 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:01:07.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:07.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:01:07.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:01:07.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.289 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:07.290 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.586028296999999, 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:07.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:01:07.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:01:07.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.581792370000002, 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.574952604000003, 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:01:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 42.820896185999999, 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:07.313 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 42.818671522000002, 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:07.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:01:07.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:01:07.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:07.319 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:07.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T16:01:07.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:01:07.322 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:07.324+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:07.422 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:07.425+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:07.505 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:07.506 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:07.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:01:07.520 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:01:07.523 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:07.525+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:07.540 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:01:07.560 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:01:07.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:01:07.601 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:01:07.621 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:01:07.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:01:07.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:01:07.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:01:07.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:01:07.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:01:07.623 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:01:07.623 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:07.626+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:07.660 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:07.660 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:07.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:01:07.756 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:07.759+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:07.826 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:07.828+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:07.854 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:07.854 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:07.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:01:07.920 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T16:01:07.921 INFO:tasks.workunit.client.0.smithi122.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-04-24T16:01:07.924 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:01:07.927+0000 7f2bdeeec700 0 -- 172.21.15.122:0/2659477248 >> [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] conn(0x7f2bc403c7e0 msgr2=0x7f2bc403ebc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:01:07.927 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:07.929+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:07.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:01:07.930 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd37497 2024-04-24T16:01:07.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T16:01:08.006 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.007 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:01:08.034 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-04-24T16:01:08.036+0000 7fbf08d0e700 0 -- 172.21.15.122:0/1682166485 >> [v2:172.21.15.32:6808/41951,v1:172.21.15.32:6809/41951] conn(0x7fbee8052290 msgr2=0x7fbee8061870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-24T16:01:08.051 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:01:08.053+0000 7fbf09d10700 0 -- 172.21.15.122:0/1682166485 >> [v2:172.21.15.32:6800/41952,v1:172.21.15.32:6801/41952] conn(0x558b26435410 msgr2=0x558b264377f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:01:08.052 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 100% complete...done. 2024-04-24T16:01:08.055 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-04-24T16:01:08.056 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns status testrbdnbd37497 2024-04-24T16:01:08.058 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:08.061+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:08.095 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-04-24T16:01:08.097 INFO:tasks.workunit.client.0.smithi122.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-04-24T16:01:08.098 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-04-24T16:01:08.129 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:08.131+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:08.183 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.184 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:01:08.234 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:08.237+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:08.331 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:08.334+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:08.356 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.356 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:01:08.435 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:08.437+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:08.490 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.490 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:01:08.536 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:08.539+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:08.599 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns status testrbdnbd37497 2024-04-24T16:01:08.599 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'Watchers: none' 2024-04-24T16:01:08.623 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.623 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:01:08.639 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:08.642+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:08.645 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-04-24T16:01:08.646 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-04-24T16:01:08.646 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd37497 2024-04-24T16:01:08.739 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:08.742+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:08.762 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.762 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:01:08.840 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:08.843+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:08.895 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:08.895 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:08.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:01:08.928 INFO:tasks.workunit.client.0.smithi122.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-04-24T16:01:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T16:01:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:01:08.935 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd37497 2024-04-24T16:01:08.941 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-04-24T16:01:08.944+0000 7f23e3ef9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 41940) UID: 0 2024-04-24T16:01:08.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T16:01:09.038 INFO:tasks.workunit.client.0.smithi122.stderr:2024-04-24T16:01:09.041+0000 7f33f58d3700 0 -- 172.21.15.122:0/1445096929 >> [v2:172.21.15.32:6808/41951,v1:172.21.15.32:6809/41951] conn(0x7f33d0003510 msgr2=0x7f33d0009500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:01:09.046 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:09.045+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:09.054 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-04-24T16:01:09.057+0000 7f33f60d4700 0 -- 172.21.15.122:0/1445096929 >> [v2:172.21.15.86:6816/41924,v1:172.21.15.86:6817/41924] conn(0x7f33dc008980 msgr2=0x7f33dc00ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T16:01:09.068 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 100% complete...done. 2024-04-24T16:01:09.071 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd namespace remove rbd/ns 2024-04-24T16:01:09.081 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:09.081 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:09.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:01:09.116 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global rm global rbd_default_pool 2024-04-24T16:01:09.142 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:09.145+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:09.146 INFO:tasks.workunit.client.0.smithi122.stderr:+ ceph osd pool delete new_default_pool37497 new_default_pool37497 --yes-i-really-really-mean-it 2024-04-24T16:01:09.217 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:09.217 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:09.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T16:01:09.243 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:09.246+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:09.344 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-04-24T16:01:09.347+0000 7f8405bb3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 41943) UID: 0 2024-04-24T16:01:09.367 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:09.367 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:09.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T16:01:09.445 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:09.447+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:09.546 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:09.549+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:09.559 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:09.560 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:09.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T16:01:09.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T16:01:09.646 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:09.649+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:09.712 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-04-24T16:01:09.712 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T16:01:09.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T16:01:09.748 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:09.751+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:09.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-04-24T16:01:09.812 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-04-24T16:01:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-04-24T16:01:09.849 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-04-24T16:01:09.852+0000 7f82a5744700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 41945) UID: 0 2024-04-24T16:01:09.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-04-24T16:01:09.866 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:09.866 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:09.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-04-24T16:01:09.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T16:01:09.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T16:01:09.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T16:01:09.930 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T16:01:09.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T16:01:09.950 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:09.953+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:10.037 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:10.037 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:10.037 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:10.037 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:10.038 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:10.051 INFO:tasks.workunit.client.0.smithi122.stderr:pool 'new_default_pool37497' does not exist 2024-04-24T16:01:10.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T16:01:10.052 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:10.053+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:10.061 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T16:01:10.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-24T16:01:10.128 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-04-24T16:01:10.129 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T16:01:10.152 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-24T16:01:10.154+0000 7f03f40b1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 41918) UID: 0 2024-04-24T16:01:10.196 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:10.196 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:10.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:10.196 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:37 3.2 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.891636285999994, 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039979718999999997, 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 37 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.305862+0000", 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310731+0000", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048686179999999999 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310750+0000", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9162000000000001e-05 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310758+0000", 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3010000000000007e-06 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.310812+0000", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3934000000000002e-05 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345835+0000", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035023501999999998 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.345842+0000", 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6889999999999998e-06 2024-04-24T16:01:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:42 3.0 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.882016222000004, 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.082334810999999994, 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 42 2024-04-24T16:01:10.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.315482+0000", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320437+0000", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049549470000000003 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320466+0000", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.866e-05 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320476+0000", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0127e-05 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.320541+0000", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5307000000000003e-05 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397814+0000", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077272708999999995 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.397817+0000", 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6200000000000001e-06 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:43 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.861615654999994, 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059108977, 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335883+0000", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.335881+0000", 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348781+0000", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01290066 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348795+0000", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3862999999999999e-05 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348801+0000", 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2879999999999998e-06 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348865+0000", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3188999999999994e-05 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349020+0000", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015495799999999999 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.349160+0000", 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014037399999999999 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.356919+0000", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077593849999999997 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394833+0000", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037913660000000002 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394905+0000", 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2242000000000004e-05 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.394992+0000", 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6860000000000005e-05 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:11 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.848683852999997, 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046329355000000003, 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.210 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348815+0000", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348814+0000", 2024-04-24T16:01:10.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348817+0000", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2480000000000001e-06 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348823+0000", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000003e-06 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348826+0000", 2024-04-24T16:01:10.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0879999999999999e-06 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.354900+0000", 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060739899999999996 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355023+0000", 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012289199999999999 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395021+0000", 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039997924999999997 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395090+0000", 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9014e-05 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395144+0000", 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4549999999999998e-05 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:12 1.3 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.848650101999993, 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050992779000000002, 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348849+0000", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348848+0000", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348851+0000", 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348861+0000", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8730000000000002e-06 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.348866+0000", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9110000000000001e-06 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355064+0000", 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061979590000000003 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355167+0000", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102821 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395162+0000", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039995267000000001 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.395208+0000", 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5955999999999998e-05 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.399841+0000", 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046332650000000001 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:49 3.2 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.847127692000001, 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.052762045, 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.219 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 49 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350371+0000", 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.350370+0000", 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355723+0000", 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053531719999999998 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355757+0000", 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4445e-05 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355766+0000", 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0650000000000005e-06 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355813+0000", 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6224999999999999e-05 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403131+0000", 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047318305999999997 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403133+0000", 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1969999999999999e-06 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:50 3.2 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.223 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.841716575999996, 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.067965833000000003, 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355782+0000", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.355781+0000", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365607+0000", 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0098262650000000007 2024-04-24T16:01:10.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365632+0000", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4953999999999999e-05 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365642+0000", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9369999999999999e-06 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365727+0000", 2024-04-24T16:01:10.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4645e-05 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423742+0000", 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058015588999999999 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423748+0000", 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7030000000000003e-06 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:45 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.833968057999996, 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058712762000000002, 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363531+0000", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.363528+0000", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999971 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381287+0000", 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017758848000000001 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381322+0000", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5343999999999999e-05 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381332+0000", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381404+0000", 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2154000000000001e-05 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381613+0000", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020882299999999999 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381745+0000", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132385 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.388888+0000", 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071426470000000002 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422154+0000", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033266477000000003 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422195+0000", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0877000000000002e-05 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422243+0000", 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8495000000000002e-05 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.232 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:51 3.0 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.831838035000004, 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.049831568, 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 51 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365661+0000", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.365659+0000", 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:10.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376517+0000", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010858055 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376548+0000", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1278000000000001e-05 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376559+0000", 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-04-24T16:01:10.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.376673+0000", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011406899999999999 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415488+0000", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038815032999999999 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415492+0000", 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8319999999999999e-06 2024-04-24T16:01:10.236 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:14 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.816135078000002, 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041092273999999998, 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:10.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381364+0000", 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381347+0000", 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999833 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381370+0000", 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.247e-05 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381379+0000", 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1689999999999998e-06 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381384+0000", 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7520000000000002e-06 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385487+0000", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041029660000000004 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385622+0000", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134466 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422299+0000", 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036676942999999997 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422396+0000", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6745999999999998e-05 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422456+0000", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0340000000000002e-05 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(osd.4.11:15 1.3 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:10.241 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.816107091000006, 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047074032000000002, 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381392+0000", 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381391+0000", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381396+0000", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6859999999999997e-06 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381407+0000", 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.381415+0000", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5959999999999998e-06 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385656+0000", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042416219999999996 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.385781+0000", 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012503899999999999 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422511+0000", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.036729669999999999 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.422592+0000", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1097e-05 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.428466+0000", 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058734049999999999 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.245 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:53 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.799336014999994, 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064123911000000006, 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 53 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.246 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398163+0000", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398161+0000", 2024-04-24T16:01:10.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398172+0000", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.165e-05 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398195+0000", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2427000000000001e-05 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.398204+0000", 2024-04-24T16:01:10.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9390000000000003e-06 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402587+0000", 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043833250000000004 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.402724+0000", 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137064 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:10.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403194+0000", 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046930399999999998 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405183+0000", 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019898709999999998 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462208+0000", 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057024257000000002 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462255+0000", 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6830999999999999e-05 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462287+0000", 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2092999999999999e-05 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:54 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.251 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.793992490999997, 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058932545000000003, 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:10.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403506+0000", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403505+0000", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403512+0000", 2024-04-24T16:01:10.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7660000000000001e-06 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403523+0000", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1673e-05 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403531+0000", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7700000000000001e-06 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403646+0000", 2024-04-24T16:01:10.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.403716+0000", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9858999999999996e-05 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.404093+0000", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037685600000000001 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.405212+0000", 2024-04-24T16:01:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001119051 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462334+0000", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057121986999999999 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462394+0000", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462439+0000", 2024-04-24T16:01:10.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4336999999999999e-05 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:55 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.781782246000006, 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046861273000000002, 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415716+0000", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415721+0000", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4820000000000002e-06 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415736+0000", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5413e-05 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415743+0000", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8950000000000001e-06 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415795+0000", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1944999999999998e-05 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.415874+0000", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8633000000000004e-05 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:10.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.416161+0000", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028751299999999998 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.418445+0000", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002283949 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462521+0000", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044075240000000002 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462551+0000", 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0049999999999999e-05 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462578+0000", 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7101999999999999e-05 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4762.0:56 3.0 3:081f1115:::rbd_object_map.129ad8b6ae3f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.774347081000002, 2024-04-24T16:01:10.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039572995, 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4762", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423152+0000", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423150+0000", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423157+0000", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423170+0000", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2981e-05 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423185+0000", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4290999999999999e-05 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423258+0000", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3868000000000005e-05 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423430+0000", 2024-04-24T16:01:10.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017195200000000001 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.423917+0000", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048632699999999999 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.430335+0000", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064186219999999997 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462628+0000", 2024-04-24T16:01:10.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032292416999999997 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462684+0000", 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6681999999999997e-05 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.462725+0000", 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0126999999999997e-05 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:36 3.2 3:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.590091181999995, 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048272504000000001, 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 36 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607408+0000", 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.607405+0000", 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618446+0000", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011041005 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618462+0000", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5509e-05 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618469+0000", 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618530+0000", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1189e-05 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655677+0000", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037146924999999997 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.655680+0000", 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.897e-06 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.271 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:40 1.3 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.585855254999998, 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041734325000000003, 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611643+0000", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.611640+0000", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999967 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622066+0000", 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010426151999999999 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622095+0000", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8433999999999999e-05 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622110+0000", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4705000000000001e-05 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622203+0000", 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3066000000000006e-05 2024-04-24T16:01:10.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622438+0000", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000235438 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.622647+0000", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020941699999999999 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.628491+0000", 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058436069999999998 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653255+0000", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024764194999999999 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653326+0000", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0370999999999994e-05 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.653378+0000", 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2163000000000002e-05 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.276 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.4872.0:38 3.0 3:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.579015489, 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040502996999999999, 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.4872", 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 38 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.277 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618483+0000", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.618482+0000", 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626008+0000", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007526067 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626029+0000", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0962e-05 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626045+0000", 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5483000000000001e-05 2024-04-24T16:01:10.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.626123+0000", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8016999999999995e-05 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658982+0000", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.032859897999999998 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T15:59:48.658986+0000", 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7670000000000001e-06 2024-04-24T16:01:10.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:92 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [watch reconnect cookie 140551731119520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.824959071000002, 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0109393099999999, 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92 2024-04-24T16:01:10.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372540+0000", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372537+0000", 2024-04-24T16:01:10.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372547+0000", 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7769999999999997e-06 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372567+0000", 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9355e-05 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372577+0000", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0492999999999999e-05 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372649+0000", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2008999999999994e-05 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.372660+0000", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0662999999999999e-05 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375549+0000", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889506 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375561+0000", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1314999999999999e-05 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381479+0000", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0059185719999999 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381676+0000", 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019724099999999999 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.383479+0000", 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018025129999999999 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.5559.0:93 3.0 3:372fdc5e:::rbd_header.129ad8b6ae3f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.822734406999999, 2024-04-24T16:01:10.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0108940740000001, 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.5559", 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.122:0/973256314", 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 93 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374764+0000", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374762+0000", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374773+0000", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374787+0000", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4833e-05 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.374797+0000", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3249999999999997e-06 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375581+0000", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078448500000000002 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-04-24T16:01:10.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:24.375587+0000", 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0700000000000003e-06 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.381973+0000", 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0063852769999999 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382075+0000", 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102439 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382503+0000", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042781200000000002 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385244+0000", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027411900000000001 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385556+0000", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000311611 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385612+0000", 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6409999999999997e-05 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385658+0000", 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6329000000000003e-05 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.292 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:10.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T16:01:10.294 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:10.296+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:10.395 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-04-24T16:01:10.398+0000 7fdf70fce700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 41916) UID: 0 2024-04-24T16:01:10.433 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:10.433 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:10.433 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-04-24T16:01:10.433 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:10.439 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:10.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T16:01:10.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T16:01:10.465 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T16:01:10.465 DEBUG:teuthology.orchestra.run.smithi122:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-24T16:01:10.496 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-24T16:01:10.498+0000 7fa3ea403700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 41921) UID: 0 2024-04-24T16:01:10.533 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T16:01:10.533 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-24T16:01:10.543 INFO:tasks.thrashosds:joining thrashosds 2024-04-24T16:01:10.582 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-04-24T16:01:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-04-24T16:01:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-04-24T16:01:10.583 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:10.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T16:01:10.734 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(3.0 66/65 [PushOp(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head, version: 57'209, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head@57'209, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.302403728999998, 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022039500000000001, 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433580+0000", 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.433578+0000", 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434364+0000", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078602499999999998 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434388+0000", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.419e-05 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434402+0000", 2024-04-24T16:01:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3550000000000001e-05 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434466+0000", 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4789000000000006e-05 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.434473+0000", 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0299999999999996e-06 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:23.435784+0000", 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013100329999999999 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5559.0:93 3.0 e68/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 68'273, mlcod=0'0)", 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 45.353360492, 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027935809999999998, 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382623+0000", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382622+0000", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382632+0000", 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0172e-05 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382647+0000", 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4598e-05 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382655+0000", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382703+0000", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7895999999999997e-05 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.382726+0000", 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3006999999999998e-05 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385363+0000", 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026376110000000002 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:25.385416+0000", 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3261999999999999e-05 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:76 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'296, mlcod=80'294)", 2024-04-24T16:01:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.404767359000001, 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055479819999999999, 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331216+0000", 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331215+0000", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331221+0000", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331233+0000", 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1596999999999999e-05 2024-04-24T16:01:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331240+0000", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8920000000000003e-06 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331282+0000", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1681999999999998e-05 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.331301+0000", 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9613999999999999e-05 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336737+0000", 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054352849999999998 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336764+0000", 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7348e-05 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:77 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'297, mlcod=80'294)", 2024-04-24T16:01:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.403865412999998, 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046695269999999997, 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332118+0000", 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332117+0000", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332125+0000", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332139+0000", 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3392e-05 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332146+0000", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3050000000000004e-06 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332193+0000", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6969999999999999e-05 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332216+0000", 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3156e-05 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336766+0000", 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045501680000000003 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336787+0000", 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1287000000000001e-05 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "pg_update_log_missing(3.0 epoch 80/67 rep_tid 482 entries 80'298 (0'0) error 3:372fdc5e:::rbd_header.129ad8b6ae3f:head by client.5285.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 80'294)", 2024-04-24T16:01:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.403069382999998, 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038821250000000002, 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332914+0000", 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332913+0000", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332923+0000", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9159999999999996e-06 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332940+0000", 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6986e-05 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.332949+0000", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8289999999999997e-06 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333006+0000", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6969999999999998e-05 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336796+0000", 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003790093 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:79 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'299, mlcod=80'294)", 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.402545364000002, 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003383718, 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333438+0000", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333437+0000", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333444+0000", 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4290000000000004e-06 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333455+0000", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0828e-05 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333461+0000", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4639999999999997e-06 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333503+0000", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2086999999999999e-05 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.333523+0000", 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0094000000000001e-05 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336797+0000", 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032740569999999999 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.336822+0000", 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4691000000000001e-05 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:82 3.0 e80/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 80'301, mlcod=80'294)", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.401739120999999, 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003048916, 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334244+0000", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334243+0000", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334250+0000", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1530000000000002e-06 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334260+0000", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8810000000000008e-06 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334264+0000", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4410000000000003e-06 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334305+0000", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0880999999999997e-05 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.334336+0000", 2024-04-24T16:01:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1554999999999999e-05 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337270+0000", 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002933546 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.337293+0000", 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.315e-05 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:84 3.0 e80/67 3:081f1115:::rbd_object_map.129ad8b6ae3f:head v 80'303, mlcod=80'299)", 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.395726101000001, 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044460719999999997, 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340257+0000", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340256+0000", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340263+0000", 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340274+0000", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0509000000000001e-05 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340281+0000", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7290000000000003e-06 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340339+0000", 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8041000000000002e-05 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.340361+0000", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1892e-05 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344658+0000", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042975230000000001 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.344703+0000", 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:85 3.0 e80/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 80'305, mlcod=80'299)", 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.394950100999999, 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042239729999999998, 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341033+0000", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341038+0000", 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9869999999999996e-06 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341054+0000", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5784e-05 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341071+0000", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341122+0000", 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.109e-05 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.341142+0000", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9956999999999999e-05 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345218+0000", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040761019999999999 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.345257+0000", 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8852e-05 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:101 3.0 e80/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 80'318, mlcod=80'314)", 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.379482842000002, 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041012970000000003, 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356500+0000", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356507+0000", 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0140000000000001e-06 2024-04-24T16:01:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356517+0000", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.005e-05 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356523+0000", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4489999999999996e-06 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356566+0000", 2024-04-24T16:01:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2168e-05 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.356582+0000", 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6711e-05 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360572+0000", 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039900580000000003 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.360602+0000", 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9408e-05 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:102 3.0 e80/67 3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 80'320, mlcod=80'314)", 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.373795901000001, 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044501560000000003, 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362187+0000", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362193+0000", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0310000000000004e-06 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362203+0000", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0345e-05 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362208+0000", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9300000000000002e-06 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362246+0000", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8188000000000001e-05 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.362259+0000", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3057999999999999e-05 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366606+0000", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043471830000000001 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.366638+0000", 2024-04-24T16:01:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1063999999999999e-05 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5285.0:107 3.0 e80/67 3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 80'325, mlcod=80'320)", 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.365933221999999, 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045367460000000004, 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370050+0000", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370049+0000", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370055+0000", 2024-04-24T16:01:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2560000000000002e-06 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370064+0000", 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1910000000000006e-06 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370068+0000", 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.267e-06 2024-04-24T16:01:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370105+0000", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6418999999999998e-05 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.370130+0000", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5082000000000001e-05 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374558+0000", 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004428871 2024-04-24T16:01:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:00:44.374587+0000", 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.832e-05 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:703 3.0 e93/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:7 v 93'466, mlcod=92'463)", 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:05.906014+0000", 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.8299691630000003, 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024114100000000001, 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906014+0000", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906014+0000", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906014+0000", 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906023+0000", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.149e-06 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906048+0000", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4836000000000001e-05 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.906060+0000", 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2687999999999999e-05 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.907131+0000", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010705529999999999 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.907161+0000", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9848999999999998e-05 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.908381+0000", 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012203299999999999 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.908426+0000", 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4456000000000003e-05 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:711 3.0 e93/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:7 v 93'475, mlcod=93'471)", 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:05.913405+0000", 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.8225779580000001, 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040384499999999999, 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913405+0000", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913405+0000", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913404+0000", 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913414+0000", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9510000000000001e-06 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913431+0000", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6935e-05 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913439+0000", 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1580000000000005e-06 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913544+0000", 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010458099999999999 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.913565+0000", 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1429000000000001e-05 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.917387+0000", 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038213610000000001 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:05.917444+0000", 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7027999999999997e-05 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:717 3.0 e94/67 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:7 v 94'478, mlcod=93'475)", 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.907542+0000", 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8284408839999999, 2024-04-24T16:01:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047360429999999997, 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907542+0000", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907542+0000", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907542+0000", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907549+0000", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8870000000000003e-06 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907582+0000", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3348e-05 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907593+0000", 2024-04-24T16:01:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0321e-05 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907651+0000", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8159999999999999e-05 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.907676+0000", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5500999999999999e-05 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912234+0000", 2024-04-24T16:01:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045580129999999996 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912278+0000", 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4147000000000003e-05 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:720 3.0 e94/67 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:7 v 94'480, mlcod=93'475)", 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.909962+0000", 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8260217409999999, 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032206320000000002, 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909962+0000", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909962+0000", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909960+0000", 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909972+0000", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1483e-05 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909984+0000", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1556000000000001e-05 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.909992+0000", 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0160000000000008e-06 2024-04-24T16:01:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.910419+0000", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042711499999999999 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.910443+0000", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4600000000000002e-05 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.913144+0000", 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027005340000000001 2024-04-24T16:01:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.913182+0000", 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8405999999999998e-05 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.5666.0:56 3.0 e94/67 3:372fdc5e:::rbd_header.129ad8b6ae3f:head v 94'481, mlcod=93'475)", 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.912641+0000", 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8233425140000001, 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031018959999999998, 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912641+0000", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912641+0000", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912640+0000", 2024-04-24T16:01:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912648+0000", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6539999999999999e-06 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912663+0000", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5398999999999999e-05 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912670+0000", 2024-04-24T16:01:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3799999999999996e-06 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912789+0000", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011809199999999999 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.912808+0000", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9896999999999999e-05 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.915698+0000", 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002889328 2024-04-24T16:01:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.915743+0000", 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4879000000000002e-05 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:724 3.0 e94/67 3:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:7 v 94'483, mlcod=94'478)", 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.916882+0000", 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.819101694, 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035817900000000001, 2024-04-24T16:01:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916882+0000", 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916882+0000", 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916880+0000", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916891+0000", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0321e-05 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916906+0000", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5948e-05 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916914+0000", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1919999999999999e-06 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.916980+0000", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6217999999999995e-05 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.917004+0000", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4034000000000001e-05 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.920419+0000", 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034153460000000001 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.920463+0000", 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4137000000000001e-05 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:725 3.0 e94/67 3:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:7 v 94'485, mlcod=94'478)", 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.918619+0000", 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.817364296, 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025495130000000002, 2024-04-24T16:01:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918619+0000", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918619+0000", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918618+0000", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.999999 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918626+0000", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5310000000000001e-06 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918636+0000", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8430000000000005e-06 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918643+0000", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1029999999999996e-06 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918680+0000", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6192999999999997e-05 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.918699+0000", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9440999999999999e-05 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.921128+0000", 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002428627 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.921169+0000", 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0852000000000001e-05 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:728 3.0 e94/67 3:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:7 v 94'487, mlcod=94'483)", 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-04-24T16:01:06.923763+0000", 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.8122200680000002, 2024-04-24T16:01:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002662817, 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923763+0000", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923763+0000", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-04-24T16:01:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923762+0000", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923772+0000", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0225e-05 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923790+0000", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7586000000000001e-05 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923798+0000", 2024-04-24T16:01:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8970000000000008e-06 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923847+0000", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9852999999999999e-05 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.923870+0000", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2328e-05 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.926377+0000", 2024-04-24T16:01:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025071249999999998 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-04-24T16:01:06.926426+0000", 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9128000000000001e-05 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T16:01:10.815 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:10.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T16:01:10.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T16:01:10.953 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:10.953 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:10.953 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:10.953 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:10.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T16:01:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T16:01:11.065 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T16:01:11.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:11.087 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.087 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:11.087 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:11.088 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:11.088 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T16:01:11.223 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.223 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:11.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:11.223 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:11.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:37 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.913734204999997, 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033570197000000003, 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311061+0000", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.311059+0000", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318987+0000", 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079271020000000001 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319006+0000", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9423e-05 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319013+0000", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0369999999999997e-06 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2231999999999996e-05 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319221+0000", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014542600000000001 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319439+0000", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021874399999999999 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.336002+0000", 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016562265 2024-04-24T16:01:11.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344455+0000", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084530539999999998 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344497+0000", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2246999999999997e-05 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344631+0000", 2024-04-24T16:01:11.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134382 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:26 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.905768175000006, 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025660013999999998, 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:11.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319027+0000", 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319026+0000", 2024-04-24T16:01:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319030+0000", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5430000000000004e-06 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319037+0000", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8419999999999999e-06 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319043+0000", 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9259999999999997e-06 2024-04-24T16:01:11.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324420+0000", 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053767219999999996 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324514+0000", 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3728999999999997e-05 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344676+0000", 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020162178999999999 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344687+0000", 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1379000000000001e-05 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:27 1.6 1:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.233 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.905733979999994, 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025650549000000002, 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:11.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319061+0000", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319065+0000", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7740000000000002e-06 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319071+0000", 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-04-24T16:01:11.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319075+0000", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2640000000000002e-06 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324538+0000", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054631139999999998 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324591+0000", 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2939999999999998e-05 2024-04-24T16:01:11.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344706+0000", 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020115052000000001 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.344712+0000", 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.237 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:40 1.0 1:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.905715341999993, 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026475325000000001, 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:11.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.319080+0000", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334458+0000", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015378644 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334474+0000", 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6113999999999999e-05 2024-04-24T16:01:11.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334482+0000", 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1099999999999997e-06 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334524+0000", 2024-04-24T16:01:11.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2202e-05 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334642+0000", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011854 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.334793+0000", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015110300000000001 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345213+0000", 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010419182000000001 2024-04-24T16:01:11.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345477+0000", 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026476499999999998 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345518+0000", 2024-04-24T16:01:11.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0911000000000002e-05 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345555+0000", 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7051000000000002e-05 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:39 1.7 e44/34 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.898392604999998, 2024-04-24T16:01:11.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018712814000000001, 2024-04-24T16:01:11.244 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.244 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.244 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326403+0000", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326402+0000", 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327404+0000", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010010990000000001 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327420+0000", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6398000000000001e-05 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327427+0000", 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5120000000000002e-06 2024-04-24T16:01:11.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327477+0000", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9120999999999999e-05 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.327505+0000", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7936000000000002e-05 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345069+0000", 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017564079 2024-04-24T16:01:11.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345116+0000", 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6990000000000002e-05 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:42 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.248 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.876733582, 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033616607999999999, 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348062+0000", 2024-04-24T16:01:11.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.348061+0000", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352265+0000", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042039180000000001 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352284+0000", 2024-04-24T16:01:11.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8915000000000001e-05 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352294+0000", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.354985+0000", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026904910000000001 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355147+0000", 2024-04-24T16:01:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016283000000000001 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.355332+0000", 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018438600000000001 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360923+0000", 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055916150000000003 2024-04-24T16:01:11.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381529+0000", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020605826000000001 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381590+0000", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1080000000000005e-05 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381678+0000", 2024-04-24T16:01:11.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8090000000000005e-05 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:32 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:snapdir [list-snaps] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.872484947999993, 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029559515000000001, 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352310+0000", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352309+0000", 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352316+0000", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4200000000000001e-06 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352325+0000", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5350000000000004e-06 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352330+0000", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2349999999999999e-06 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359396+0000", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070663979999999998 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359532+0000", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000135938 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381736+0000", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203602999999999 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381816+0000", 2024-04-24T16:01:11.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9589e-05 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381870+0000", 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4367000000000003e-05 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:33 1.0 1:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head [copy-get max 8388608] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.872450430000001, 2024-04-24T16:01:11.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037891718999999997, 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352345+0000", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352344+0000", 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352349+0000", 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3200000000000001e-06 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352354+0000", 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2630000000000003e-06 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352361+0000", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3930000000000002e-06 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359573+0000", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007211439 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.359682+0000", 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109233 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381885+0000", 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022203005000000001 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.381941+0000", 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6447999999999999e-05 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.390237+0000", 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082951980000000002 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:51 1.4 e44/12 1:37c16acd:::rbd_data.129ad8b6ae3f.0000000000000009:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:11.263 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.842289326, 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032431422000000001, 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382506+0000", 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382505+0000", 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393162+0000", 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010656453999999999 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393204+0000", 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2818000000000002e-05 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393213+0000", 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5390000000000007e-06 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393267+0000", 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4101999999999998e-05 2024-04-24T16:01:11.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393298+0000", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0601000000000001e-05 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414886+0000", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021588782000000001 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414937+0000", 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1081000000000001e-05 2024-04-24T16:01:11.267 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:58 1.4 e44/12 1:3560594c:::rbd_data.129ad8b6ae3f.000000000000000a:head v 44'5, mlcod=44'3)", 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.791194387000004, 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028265492999999999, 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:11.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433601+0000", 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.433600+0000", 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441680+0000", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080799940000000001 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441712+0000", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441724+0000", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1868000000000001e-05 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441771+0000", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6752000000000002e-05 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.441795+0000", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4091e-05 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461768+0000", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019972843000000001 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461866+0000", 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8933999999999998e-05 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:60 1.4 e44/12 1:32fa21bc:::rbd_data.129ad8b6ae3f.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.737675874999994, 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022120865999999999, 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.272 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487119+0000", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487117+0000", 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491516+0000", 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004398964 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491535+0000", 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491545+0000", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0728e-05 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491594+0000", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8121999999999998e-05 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.491621+0000", 2024-04-24T16:01:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509194+0000", 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017572577999999998 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.509240+0000", 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.638e-05 2024-04-24T16:01:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:34 1.4 e44/12 1:34e2fcb3:ns::rbd_data.1308dd266c0.0000000000000002:head v 44'8, mlcod=44'5)", 2024-04-24T16:01:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.614335714999996, 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017242904999999999, 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610460+0000", 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.610459+0000", 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617381+0000", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069222959999999997 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617412+0000", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1445999999999997e-05 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617421+0000", 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2800000000000003e-06 2024-04-24T16:01:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617457+0000", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6553000000000001e-05 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.617474+0000", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6370000000000001e-05 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627659+0000", 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010185464999999999 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627703+0000", 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3571e-05 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:38 1.0 1:0992468e:ns::rbd_data.1308dd266c0.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.597466260000004, 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031527685, 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627329+0000", 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627327+0000", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633335+0000", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060079390000000003 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633357+0000", 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1869999999999999e-05 2024-04-24T16:01:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633388+0000", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1106999999999999e-05 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633453+0000", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5019999999999998e-05 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633604+0000", 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150579 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633787+0000", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018269200000000001 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648083+0000", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014295924 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658775+0000", 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010692624 2024-04-24T16:01:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658821+0000", 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6319999999999997e-05 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.658857+0000", 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5311000000000001e-05 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:42 1.7 e44/34 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.589249835000004, 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02635374, 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635546+0000", 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635544+0000", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637062+0000", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015188179999999999 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637080+0000", 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7734999999999999e-05 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637087+0000", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0469999999999996e-06 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637145+0000", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658000000000003e-05 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.637169+0000", 2024-04-24T16:01:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4264e-05 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661863+0000", 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024693874000000001 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.661899+0000", 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6337000000000001e-05 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:45 1.7 e44/34 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.575709687, 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015482004000000001, 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649086+0000", 2024-04-24T16:01:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.649085+0000", 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650189+0000", 2024-04-24T16:01:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001104056 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650208+0000", 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9624999999999999e-05 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650215+0000", 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1849999999999998e-06 2024-04-24T16:01:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650259+0000", 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3219000000000001e-05 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650285+0000", 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5891e-05 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664536+0000", 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014251257 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664568+0000", 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1884000000000002e-05 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:46 1.0 1:039777e8:ns::rbd_data.1308dd266c0.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.573825017999994, 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01403069, 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650970+0000", 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.650968+0000", 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654804+0000", 2024-04-24T16:01:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038356699999999998 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654818+0000", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3517e-05 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654824+0000", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0929999999999999e-06 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.654874+0000", 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9947999999999997e-05 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655028+0000", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154573 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655182+0000", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015405900000000001 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.663943+0000", 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087608400000000006 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664918+0000", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097500500000000003 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664970+0000", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1439e-05 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.665001+0000", 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1538000000000002e-05 2024-04-24T16:01:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:51 1.6 1:6b109d0e:ns::rbd_data.1308dd266c0.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.553216758999994, 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021400631, 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671579+0000", 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.671577+0000", 2024-04-24T16:01:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676328+0000", 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047504269999999998 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676351+0000", 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3896999999999998e-05 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676360+0000", 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2430000000000005e-06 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676415+0000", 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5606000000000003e-05 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676536+0000", 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120541 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.676679+0000", 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014289500000000001 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.683354+0000", 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066749019999999999 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692848+0000", 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094942619999999998 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692924+0000", 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6124999999999994e-05 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692979+0000", 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5173999999999997e-05 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:56 1.7 e44/34 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head v 44'6, mlcod=44'4)", 2024-04-24T16:01:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.543295259000004, 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032740621999999997, 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681500+0000", 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681499+0000", 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682746+0000", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012476480000000001 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682772+0000", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5783e-05 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682785+0000", 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2437e-05 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682857+0000", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2779000000000002e-05 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.682886+0000", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8571000000000001e-05 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714206+0000", 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031319597999999997 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714241+0000", 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5092999999999998e-05 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:54 1.4 e44/12 1:3ebef24b:ns::rbd_data.1308dd266c0.000000000000000a:head v 44'9, mlcod=44'7)", 2024-04-24T16:01:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.538647001000001, 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029832606000000001, 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686148+0000", 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686147+0000", 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692622+0000", 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064747600000000004 2024-04-24T16:01:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692637+0000", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4902000000000001e-05 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692650+0000", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3138e-05 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692702+0000", 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1487999999999999e-05 2024-04-24T16:01:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.692726+0000", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4799000000000001e-05 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715948+0000", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023221434999999999 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715981+0000", 2024-04-24T16:01:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3147e-05 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:58 1.7 e44/34 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head v 44'7, mlcod=44'4)", 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.53715321, 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028194987000000001, 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687642+0000", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687641+0000", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689042+0000", 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014007920000000001 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689070+0000", 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8291000000000001e-05 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689082+0000", 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2204999999999999e-05 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689133+0000", 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1218000000000003e-05 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689165+0000", 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1665999999999998e-05 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715801+0000", 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026636263 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715837+0000", 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5717999999999999e-05 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.323 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T16:01:11.422 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:11.424 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:11.436 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-24T16:01:11.050071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229821755,"num_objects":116,"num_object_clones":17,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":22,"num_read":7392,"num_read_kb":178073,"num_write":1057,"num_write_kb":273520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1163,"ondisk_log_size":1163,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2246504,"kb_used_data":442872,"kb_used_omap":9,"kb_used_meta":1800438,"kb_avail":563984536,"statfs":{"total":579820584960,"available":577520164864,"internally_reserved":0,"allocated":453500928,"data_stored":450413940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843648528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-70,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-38,"num_read_kb":-30,"num_write":-6,"num_write_kb":-6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-35,"num_keys_recovered":-2,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.272361"},"pg_stats":[{"pgid":"1.5","version":"83'6","reported_seq":146,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.913081+0000","last_change":"2024-04-24T16:01:05.913273+0000","last_active":"2024-04-24T16:01:05.913081+0000","last_peered":"2024-04-24T16:01:05.913081+0000","last_clean":"2024-04-24T16:01:05.913081+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:05.913081+0000","last_undegraded":"2024-04-24T16:01:05.913081+0000","last_fullsized":"2024-04-24T16:01:05.913081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000445167,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.4","version":"44'9","reported_seq":185,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.904687+0000","last_change":"2024-04-24T16:01:05.905010+0000","last_active":"2024-04-24T16:01:05.904687+0000","last_peered":"2024-04-24T16:01:05.904687+0000","last_clean":"2024-04-24T16:01:05.904687+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:05.904687+0000","last_undegraded":"2024-04-24T16:01:05.904687+0000","last_fullsized":"2024-04-24T16:01:05.904687+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000551327,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.3","version":"81'6","reported_seq":126,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.904635+0000","last_change":"2024-04-24T16:01:05.905082+0000","last_active":"2024-04-24T16:01:05.904635+0000","last_peered":"2024-04-24T16:01:05.904635+0000","last_clean":"2024-04-24T16:01:05.904635+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:05.904635+0000","last_undegraded":"2024-04-24T16:01:05.904635+0000","last_fullsized":"2024-04-24T16:01:05.904635+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085892599999999998,"stat_sum":{"num_bytes":16777254,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"2.0","version":"13'117","reported_seq":261,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.005066+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:05.005066+0000","last_peered":"2024-04-24T16:01:05.005066+0000","last_clean":"2024-04-24T16:01:05.005066+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:05.005066+0000","last_undegraded":"2024-04-24T16:01:05.005066+0000","last_fullsized":"2024-04-24T16:01:05.005066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"93'122","reported_seq":605,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.909872+0000","last_change":"2024-04-24T16:01:05.910172+0000","last_active":"2024-04-24T16:01:05.909872+0000","last_peered":"2024-04-24T16:01:05.909872+0000","last_clean":"2024-04-24T16:01:05.909872+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:05.909872+0000","last_undegraded":"2024-04-24T16:01:05.909872+0000","last_fullsized":"2024-04-24T16:01:05.909872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035137929999999999,"stat_sum":{"num_bytes":13111334,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":3,"num_read":590,"num_read_kb":11171,"num_write":106,"num_write_kb":8520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.2","version":"50'8","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912801+0000","last_change":"2024-04-24T16:01:05.913002+0000","last_active":"2024-04-24T16:01:05.912801+0000","last_peered":"2024-04-24T16:01:05.912801+0000","last_clean":"2024-04-24T16:01:05.912801+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:05.912801+0000","last_undegraded":"2024-04-24T16:01:05.912801+0000","last_fullsized":"2024-04-24T16:01:05.912801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326137,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.0","version":"93'476","reported_seq":1831,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.924902+0000","last_change":"2024-04-24T16:01:05.918697+0000","last_active":"2024-04-24T16:01:05.924902+0000","last_peered":"2024-04-24T16:01:05.924902+0000","last_clean":"2024-04-24T16:01:05.924902+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:05.924902+0000","last_undegraded":"2024-04-24T16:01:05.924902+0000","last_fullsized":"2024-04-24T16:01:05.924902+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:07.896694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014121909,"stat_sum":{"num_bytes":26224678,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":4372,"num_read_kb":48797,"num_write":436,"num_write_kb":41990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.0","version":"83'6","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.963320+0000","last_change":"2024-04-24T16:01:05.963470+0000","last_active":"2024-04-24T16:01:05.963320+0000","last_peered":"2024-04-24T16:01:05.963320+0000","last_clean":"2024-04-24T16:01:05.963320+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:05.963320+0000","last_undegraded":"2024-04-24T16:01:05.963320+0000","last_fullsized":"2024-04-24T16:01:05.963320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023370199999999999,"stat_sum":{"num_bytes":16777332,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.2","version":"93'258","reported_seq":1237,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.926141+0000","last_change":"2024-04-24T16:01:05.924135+0000","last_active":"2024-04-24T16:01:05.926141+0000","last_peered":"2024-04-24T16:01:05.926141+0000","last_clean":"2024-04-24T16:01:05.926141+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:05.926141+0000","last_undegraded":"2024-04-24T16:01:05.926141+0000","last_fullsized":"2024-04-24T16:01:05.926141+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.019641927999999999,"stat_sum":{"num_bytes":29454355,"num_objects":21,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":10,"num_read":909,"num_read_kb":64611,"num_write":210,"num_write_kb":56007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":100,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912422+0000","last_change":"2024-04-24T16:01:05.912642+0000","last_active":"2024-04-24T16:01:05.912422+0000","last_peered":"2024-04-24T16:01:05.912422+0000","last_clean":"2024-04-24T16:01:05.912422+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:05.912422+0000","last_undegraded":"2024-04-24T16:01:05.912422+0000","last_fullsized":"2024-04-24T16:01:05.912422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037122199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.3","version":"93'139","reported_seq":1281,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.924024+0000","last_change":"2024-04-24T16:01:05.924024+0000","last_active":"2024-04-24T16:01:05.924024+0000","last_peered":"2024-04-24T16:01:05.924024+0000","last_clean":"2024-04-24T16:01:05.924024+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:05.924024+0000","last_undegraded":"2024-04-24T16:01:05.924024+0000","last_fullsized":"2024-04-24T16:01:05.924024+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:29:42.919951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011000796,"stat_sum":{"num_bytes":26222699,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":1282,"num_read_kb":48586,"num_write":89,"num_write_kb":33363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.6","version":"83'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.963381+0000","last_change":"2024-04-24T16:01:05.963550+0000","last_active":"2024-04-24T16:01:05.963381+0000","last_peered":"2024-04-24T16:01:05.963381+0000","last_clean":"2024-04-24T16:01:05.963381+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:05.963381+0000","last_undegraded":"2024-04-24T16:01:05.963381+0000","last_fullsized":"2024-04-24T16:01:05.963381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026120899999999999,"stat_sum":{"num_bytes":8388800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":7,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.7","version":"83'9","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912604+0000","last_change":"2024-04-24T16:01:05.912808+0000","last_active":"2024-04-24T16:01:05.912604+0000","last_peered":"2024-04-24T16:01:05.912604+0000","last_clean":"2024-04-24T16:01:05.912604+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:05.912604+0000","last_undegraded":"2024-04-24T16:01:05.912604+0000","last_fullsized":"2024-04-24T16:01:05.912604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031459499999999998,"stat_sum":{"num_bytes":25165955,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":95013066,"num_objects":64,"num_object_clones":17,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":22,"num_read":7153,"num_read_kb":173165,"num_write":841,"num_write_kb":139880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177143808,"data_stored":178033044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":995,"ondisk_log_size":995,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218321,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":123,"num_read_kb":4808,"num_write":57,"num_write_kb":131224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268533760,"data_stored":268436642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367944,"kb_used_data":69552,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94003896,"statfs":{"total":96636764160,"available":96259989504,"internally_reserved":0,"allocated":71221248,"data_stored":71024676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.372}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.249}]}]},{"osd":4,"up_from":11,"seq":47244640289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377148,"kb_used_data":75692,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93994692,"statfs":{"total":96636764160,"available":96250564608,"internally_reserved":0,"allocated":77508608,"data_stored":76836938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.374}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.40699999999999997}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.35099999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.435}]}]},{"osd":3,"up_from":11,"seq":47244640290,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348184,"kb_used_data":50824,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94023656,"statfs":{"total":96636764160,"available":96280223744,"internally_reserved":0,"allocated":52043776,"data_stored":51378691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.54600000000000004}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65200000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.442},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365352,"kb_used_data":63900,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94006488,"statfs":{"total":96636764160,"available":96262643712,"internally_reserved":0,"allocated":65433600,"data_stored":64762383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.313},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.44}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.505},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45800000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40500000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389456,"kb_used_data":88000,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93982384,"statfs":{"total":96636764160,"available":96237961216,"internally_reserved":0,"allocated":90112000,"data_stored":89420058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.248},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35599999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.39700000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.438},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47199999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640290,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398420,"kb_used_data":94904,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93973420,"statfs":{"total":96636764160,"available":96228782080,"internally_reserved":0,"allocated":97181696,"data_stored":96991194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.308},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36899999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.41999999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37769216,"data_stored":37748983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50356224,"data_stored":50331956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41959424,"data_stored":41943209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54120448,"data_stored":54585401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25972736,"data_stored":25966630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28147712,"data_stored":28618771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:11.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:11.451 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.451 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:11.451 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:11.451 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T16:01:11.589 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:11.590 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:11.590 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:11.590 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T16:01:11.725 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.725 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:11.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:11.725 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:35 1.2 1:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.420123082999993, 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029735411, 2024-04-24T16:01:11.726 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:11.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306202+0000", 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.306201+0000", 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318394+0000", 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012192579 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318406+0000", 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2252000000000001e-05 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318411+0000", 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.853e-06 2024-04-24T16:01:11.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318451+0000", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0348000000000001e-05 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318546+0000", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5006000000000005e-05 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.318641+0000", 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4588999999999998e-05 2024-04-24T16:01:11.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.330607+0000", 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011966152000000001 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335812+0000", 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052048069999999997 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335867+0000", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5242999999999999e-05 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335938+0000", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0689e-05 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 3.3 3:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:11.733 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.418976297, 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038257521000000003, 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 39 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:11.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307349+0000", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.307348+0000", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323147+0000", 2024-04-24T16:01:11.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015798935 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323169+0000", 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1461999999999999e-05 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323182+0000", 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3424000000000001e-05 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323249+0000", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7230000000000005e-05 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345604+0000", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022354083 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345607+0000", 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2279999999999998e-06 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:43 3.3 3:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.403124175000002, 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072225187999999996, 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 43 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.738 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323201+0000", 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.323200+0000", 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335236+0000", 2024-04-24T16:01:11.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012036696 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335258+0000", 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1624000000000002e-05 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335269+0000", 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1245e-05 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.335324+0000", 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5121000000000003e-05 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395422+0000", 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060097273 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.395427+0000", 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9269999999999996e-06 2024-04-24T16:01:11.743 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:45 3.3 3:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.380359544000001, 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076680945, 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4762", 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/2987973851", 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45 2024-04-24T16:01:11.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345966+0000", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345965+0000", 2024-04-24T16:01:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360780+0000", 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014815277 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360798+0000", 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7445999999999999e-05 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.360807+0000", 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4299999999999995e-06 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.362565+0000", 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017578649999999999 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422644+0000", 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060079248000000002 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422647+0000", 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9000000000000002e-06 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:49 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.748 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.370241187999994, 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047006919000000001, 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356084+0000", 2024-04-24T16:01:11.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.356082+0000", 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383966+0000", 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027883715999999999 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383987+0000", 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0990999999999999e-05 2024-04-24T16:01:11.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.383993+0000", 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1040000000000003e-06 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384046+0000", 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2723999999999999e-05 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384211+0000", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000164788 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384352+0000", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014086099999999999 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396467+0000", 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0121152 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402900+0000", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064327229999999996 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402975+0000", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5591000000000002e-05 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403091+0000", 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116177 2024-04-24T16:01:11.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:36 1.2 1:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.342298072999995, 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028543227000000001, 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:11.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384027+0000", 2024-04-24T16:01:11.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384031+0000", 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.563e-06 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384034+0000", 2024-04-24T16:01:11.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4070000000000002e-06 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384036+0000", 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.266e-06 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387472+0000", 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003435226 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.387560+0000", 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8672999999999996e-05 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403256+0000", 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015696001000000001 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.403333+0000", 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6588000000000006e-05 2024-04-24T16:01:11.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412571+0000", 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092377799999999993 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:50 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.342280247000005, 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031083598, 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.761 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.384045+0000", 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397331+0000", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013286381999999999 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397346+0000", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4766000000000001e-05 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397352+0000", 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.57e-06 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397397+0000", 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5210000000000003e-05 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397527+0000", 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129827 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397652+0000", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012509399999999999 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.412643+0000", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014991541000000001 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414822+0000", 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021787859999999998 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415023+0000", 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000200266 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415129+0000", 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106402 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:30 3.0 e44/34 3:063a3a47:::rbd_data.129ad8b6ae3f.0000000000000003:head v 44'18, mlcod=44'16)", 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.332662849000002, 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028161291000000001, 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393663+0000", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.393661+0000", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.406999+0000", 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013337842000000001 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407016+0000", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7127999999999999e-05 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407022+0000", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4230000000000002e-06 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407057+0000", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4181999999999999e-05 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.407078+0000", 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1294e-05 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421781+0000", 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014702977000000001 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421824+0000", 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3228e-05 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.1.11:39 1.2 1:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.328933938000006, 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02703289, 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397392+0000", 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397391+0000", 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397394+0000", 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397398+0000", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1370000000000004e-06 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.397401+0000", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1109999999999999e-06 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402822+0000", 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00542094 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.402883+0000", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0763000000000001e-05 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415336+0000", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012453581 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.415412+0000", 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6212999999999997e-05 2024-04-24T16:01:11.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.424425+0000", 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0090120310000000002 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:32 3.0 e44/34 3:13daa51e:::rbd_data.129ad8b6ae3f.0000000000000004:head v 44'20, mlcod=44'16)", 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.303049874999999, 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038655805000000001, 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.775 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423276+0000", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423275+0000", 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442274+0000", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018998629 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442293+0000", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9550000000000001e-05 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442303+0000", 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442347+0000", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4196999999999997e-05 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.442365+0000", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8691000000000001e-05 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461856+0000", 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019490851 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.461932+0000", 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5236999999999998e-05 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:59 1.2 1:5c8905d5:::rbd_data.129ad8b6ae3f.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.246828234999995, 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031085854999999999, 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479497+0000", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479495+0000", 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487809+0000", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083131520000000007 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487835+0000", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487850+0000", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4987000000000001e-05 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.487934+0000", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4285999999999999e-05 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488190+0000", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025548900000000002 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488388+0000", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019767600000000001 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.499387+0000", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010999770000000001 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510499+0000", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011112057 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510543+0000", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3414999999999997e-05 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.510583+0000", 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0368999999999999e-05 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:63 1.5 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.246807125000004, 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028318095000000001, 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.479518+0000", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486524+0000", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070065229999999997 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486535+0000", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.081e-05 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486541+0000", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1580000000000002e-06 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486612+0000", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0989999999999996e-05 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486728+0000", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115999 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486820+0000", 2024-04-24T16:01:11.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2478000000000005e-05 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507518+0000", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020697233999999998 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507728+0000", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000210182 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507763+0000", 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5525000000000003e-05 2024-04-24T16:01:11.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507837+0000", 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139000000000006e-05 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:32 3.3 3:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.133932279999996, 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036567440999999999, 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.789 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.592393+0000", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600332+0000", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079393889999999998 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600345+0000", 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2713e-05 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600354+0000", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6390000000000004e-06 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.600391+0000", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6409999999999999e-05 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628954+0000", 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028563377000000001 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.628961+0000", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6939999999999998e-06 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:40 3.3 3:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.124514509999997, 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051811866999999998, 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:11.793 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 40 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601811+0000", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.601809+0000", 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611240+0000", 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0094307669999999996 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611259+0000", 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9176000000000001e-05 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611268+0000", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9870000000000005e-06 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.611325+0000", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6617999999999999e-05 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653620+0000", 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042295121999999997 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653623+0000", 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9079999999999999e-06 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 3.3 3:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.107998343999995, 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046158614000000001, 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.797 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 42 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618327+0000", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618325+0000", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633262+0000", 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014936929 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633281+0000", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633289+0000", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9789999999999993e-06 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633332+0000", 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2987e-05 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664483+0000", 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031151545999999999 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664486+0000", 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.469e-06 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:36 1.2 1:410f1840:ns::rbd_data.1308dd266c0.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.801 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.107441928, 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036528488999999997, 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618884+0000", 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.618882+0000", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629818+0000", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010935657 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629836+0000", 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8478000000000002e-05 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629846+0000", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6250000000000002e-06 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629893+0000", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6587e-05 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630059+0000", 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166649 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.630192+0000", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013238400000000001 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648495+0000", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018303215000000001 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655289+0000", 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067943309999999998 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655360+0000", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0730999999999998e-05 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.655412+0000", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2266999999999999e-05 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 3.3 3:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.058211524000001, 2024-04-24T16:01:11.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.04639633, 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668114+0000", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.668113+0000", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679748+0000", 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011635626 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679768+0000", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9918000000000001e-05 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679777+0000", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0219999999999993e-06 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679835+0000", 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7358999999999997e-05 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714508+0000", 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034672882000000002 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714510+0000", 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 3.3 3:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.046533697000001, 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036409720999999999, 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.4872", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.122:0/1716604523", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 58 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679792+0000", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.679791+0000", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685827+0000", 2024-04-24T16:01:11.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060368319999999998 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685852+0000", 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4206000000000002e-05 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685863+0000", 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1707e-05 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.685918+0000", 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4267000000000001e-05 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716200+0000", 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030281882999999999 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716202+0000", 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0439999999999998e-06 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:57 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.045185938000003, 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034747621999999999, 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681140+0000", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681139+0000", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689202+0000", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080632789999999996 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689222+0000", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0137e-05 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689229+0000", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1810000000000004e-06 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689277+0000", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7837000000000003e-05 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689423+0000", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145487 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689536+0000", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113651 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714447+0000", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024911150999999999 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:11.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715745+0000", 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298003 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715788+0000", 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2895000000000002e-05 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715887+0000", 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9018000000000006e-05 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.0.11:12 1.5 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.037058357999996, 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031252051000000003, 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689267+0000", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689271+0000", 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7459999999999998e-06 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689276+0000", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689279+0000", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0860000000000002e-06 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693270+0000", 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003990759 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.693354+0000", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3708999999999994e-05 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716119+0000", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022765486000000001 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716191+0000", 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1427999999999994e-05 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.720519+0000", 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043286219999999999 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:11.824 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.826 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:11.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T16:01:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:11.828 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-04-24T16:01:11.778483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229821755,"num_objects":116,"num_object_clones":17,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":22,"num_read":7392,"num_read_kb":178073,"num_write":1057,"num_write_kb":273520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1163,"ondisk_log_size":1163,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":2246504,"kb_used_data":442872,"kb_used_omap":9,"kb_used_meta":1800438,"kb_avail":563984536,"statfs":{"total":579820584960,"available":577520164864,"internally_reserved":0,"allocated":453500928,"data_stored":450413940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1843648528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-70,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-38,"num_read_kb":-30,"num_write":-6,"num_write_kb":-6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-35,"num_keys_recovered":-2,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.862101"},"pg_stats":[{"pgid":"1.5","version":"83'6","reported_seq":146,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.913081+0000","last_change":"2024-04-24T16:01:05.913273+0000","last_active":"2024-04-24T16:01:05.913081+0000","last_peered":"2024-04-24T16:01:05.913081+0000","last_clean":"2024-04-24T16:01:05.913081+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:05.913081+0000","last_undegraded":"2024-04-24T16:01:05.913081+0000","last_fullsized":"2024-04-24T16:01:05.913081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000445167,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.4","version":"44'9","reported_seq":185,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.904687+0000","last_change":"2024-04-24T16:01:05.905010+0000","last_active":"2024-04-24T16:01:05.904687+0000","last_peered":"2024-04-24T16:01:05.904687+0000","last_clean":"2024-04-24T16:01:05.904687+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:05.904687+0000","last_undegraded":"2024-04-24T16:01:05.904687+0000","last_fullsized":"2024-04-24T16:01:05.904687+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000551327,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.3","version":"81'6","reported_seq":126,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.904635+0000","last_change":"2024-04-24T16:01:05.905082+0000","last_active":"2024-04-24T16:01:05.904635+0000","last_peered":"2024-04-24T16:01:05.904635+0000","last_clean":"2024-04-24T16:01:05.904635+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:05.904635+0000","last_undegraded":"2024-04-24T16:01:05.904635+0000","last_fullsized":"2024-04-24T16:01:05.904635+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085892599999999998,"stat_sum":{"num_bytes":16777254,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"2.0","version":"13'117","reported_seq":261,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.005066+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:05.005066+0000","last_peered":"2024-04-24T16:01:05.005066+0000","last_clean":"2024-04-24T16:01:05.005066+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:05.005066+0000","last_undegraded":"2024-04-24T16:01:05.005066+0000","last_fullsized":"2024-04-24T16:01:05.005066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"93'122","reported_seq":605,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.909872+0000","last_change":"2024-04-24T16:01:05.910172+0000","last_active":"2024-04-24T16:01:05.909872+0000","last_peered":"2024-04-24T16:01:05.909872+0000","last_clean":"2024-04-24T16:01:05.909872+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:05.909872+0000","last_undegraded":"2024-04-24T16:01:05.909872+0000","last_fullsized":"2024-04-24T16:01:05.909872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035137929999999999,"stat_sum":{"num_bytes":13111334,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":3,"num_read":590,"num_read_kb":11171,"num_write":106,"num_write_kb":8520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.2","version":"50'8","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912801+0000","last_change":"2024-04-24T16:01:05.913002+0000","last_active":"2024-04-24T16:01:05.912801+0000","last_peered":"2024-04-24T16:01:05.912801+0000","last_clean":"2024-04-24T16:01:05.912801+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:05.912801+0000","last_undegraded":"2024-04-24T16:01:05.912801+0000","last_fullsized":"2024-04-24T16:01:05.912801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000326137,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":8,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.0","version":"93'476","reported_seq":1831,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.924902+0000","last_change":"2024-04-24T16:01:05.918697+0000","last_active":"2024-04-24T16:01:05.924902+0000","last_peered":"2024-04-24T16:01:05.924902+0000","last_clean":"2024-04-24T16:01:05.924902+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:05.924902+0000","last_undegraded":"2024-04-24T16:01:05.924902+0000","last_fullsized":"2024-04-24T16:01:05.924902+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:07.896694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014121909,"stat_sum":{"num_bytes":26224678,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":4372,"num_read_kb":48797,"num_write":436,"num_write_kb":41990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.0","version":"83'6","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.963320+0000","last_change":"2024-04-24T16:01:05.963470+0000","last_active":"2024-04-24T16:01:05.963320+0000","last_peered":"2024-04-24T16:01:05.963320+0000","last_clean":"2024-04-24T16:01:05.963320+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:05.963320+0000","last_undegraded":"2024-04-24T16:01:05.963320+0000","last_fullsized":"2024-04-24T16:01:05.963320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023370199999999999,"stat_sum":{"num_bytes":16777332,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":6,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.2","version":"93'258","reported_seq":1237,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.926141+0000","last_change":"2024-04-24T16:01:05.924135+0000","last_active":"2024-04-24T16:01:05.926141+0000","last_peered":"2024-04-24T16:01:05.926141+0000","last_clean":"2024-04-24T16:01:05.926141+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:05.926141+0000","last_undegraded":"2024-04-24T16:01:05.926141+0000","last_fullsized":"2024-04-24T16:01:05.926141+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.019641927999999999,"stat_sum":{"num_bytes":29454355,"num_objects":21,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":10,"num_read":909,"num_read_kb":64611,"num_write":210,"num_write_kb":56007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":100,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912422+0000","last_change":"2024-04-24T16:01:05.912642+0000","last_active":"2024-04-24T16:01:05.912422+0000","last_peered":"2024-04-24T16:01:05.912422+0000","last_clean":"2024-04-24T16:01:05.912422+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:05.912422+0000","last_undegraded":"2024-04-24T16:01:05.912422+0000","last_fullsized":"2024-04-24T16:01:05.912422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037122199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"3.3","version":"93'139","reported_seq":1281,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.924024+0000","last_change":"2024-04-24T16:01:05.924024+0000","last_active":"2024-04-24T16:01:05.924024+0000","last_peered":"2024-04-24T16:01:05.924024+0000","last_clean":"2024-04-24T16:01:05.924024+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:05.924024+0000","last_undegraded":"2024-04-24T16:01:05.924024+0000","last_fullsized":"2024-04-24T16:01:05.924024+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:29:42.919951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011000796,"stat_sum":{"num_bytes":26222699,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":1282,"num_read_kb":48586,"num_write":89,"num_write_kb":33363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.6","version":"83'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.963381+0000","last_change":"2024-04-24T16:01:05.963550+0000","last_active":"2024-04-24T16:01:05.963381+0000","last_peered":"2024-04-24T16:01:05.963381+0000","last_clean":"2024-04-24T16:01:05.963381+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:05.963381+0000","last_undegraded":"2024-04-24T16:01:05.963381+0000","last_fullsized":"2024-04-24T16:01:05.963381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026120899999999999,"stat_sum":{"num_bytes":8388800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":7,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"1.7","version":"83'9","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T16:01:05.912604+0000","last_change":"2024-04-24T16:01:05.912808+0000","last_active":"2024-04-24T16:01:05.912604+0000","last_peered":"2024-04-24T16:01:05.912604+0000","last_clean":"2024-04-24T16:01:05.912604+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:05.912604+0000","last_undegraded":"2024-04-24T16:01:05.912604+0000","last_fullsized":"2024-04-24T16:01:05.912604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031459499999999998,"stat_sum":{"num_bytes":25165955,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":9,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":95013066,"num_objects":64,"num_object_clones":17,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":22,"num_read":7153,"num_read_kb":173165,"num_write":841,"num_write_kb":139880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177143808,"data_stored":178033044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":995,"ondisk_log_size":995,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":134218321,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":123,"num_read_kb":4808,"num_write":57,"num_write_kb":131224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268533760,"data_stored":268436642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367944,"kb_used_data":69552,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94003896,"statfs":{"total":96636764160,"available":96259989504,"internally_reserved":0,"allocated":71221248,"data_stored":71024676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.372}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.249}]}]},{"osd":4,"up_from":11,"seq":47244640289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377148,"kb_used_data":75692,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93994692,"statfs":{"total":96636764160,"available":96250564608,"internally_reserved":0,"allocated":77508608,"data_stored":76836938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.374}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.40699999999999997}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.35099999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.435}]}]},{"osd":3,"up_from":11,"seq":47244640290,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348184,"kb_used_data":50824,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94023656,"statfs":{"total":96636764160,"available":96280223744,"internally_reserved":0,"allocated":52043776,"data_stored":51378691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.54600000000000004}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65200000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.442},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365352,"kb_used_data":63900,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94006488,"statfs":{"total":96636764160,"available":96262643712,"internally_reserved":0,"allocated":65433600,"data_stored":64762383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.313},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.44}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29599999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.505},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45800000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40500000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389456,"kb_used_data":88000,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93982384,"statfs":{"total":96636764160,"available":96237961216,"internally_reserved":0,"allocated":90112000,"data_stored":89420058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.248},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35599999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.39700000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.438},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47199999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640290,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398420,"kb_used_data":94904,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93973420,"statfs":{"total":96636764160,"available":96228782080,"internally_reserved":0,"allocated":97181696,"data_stored":96991194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.308},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36899999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.41999999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37769216,"data_stored":37748983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50356224,"data_stored":50331956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41951232,"data_stored":41943156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41959424,"data_stored":41943209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54120448,"data_stored":54585401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25972736,"data_stored":25966630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34451456,"data_stored":34431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28147712,"data_stored":28618771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:11.829 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T16:01:11.829 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T16:01:11.829 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T16:01:11.829 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T16:01:11.830 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T16:01:11.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T16:01:11.960 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.960 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-04-24T16:01:11.960 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.961 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:11.961 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:11.961 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-04-24T16:01:11.961 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:11.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T16:01:12.057 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-04-24T16:01:12.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T16:01:12.095 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:12.096 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-04-24T16:01:12.096 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-04-24T16:01:12.096 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-04-24T16:01:12.096 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:12.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:39 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.908476718000003, 2024-04-24T16:01:12.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021404187000000002, 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324063+0000", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.324061+0000", 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325821+0000", 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017595569999999999 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325842+0000", 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1404999999999999e-05 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325853+0000", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611000000000001e-05 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325914+0000", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1773000000000006e-05 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326074+0000", 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015933599999999999 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.326261+0000", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018693599999999999 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.341385+0000", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015124576000000001 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345336+0000", 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039506599999999999 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345403+0000", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7308e-05 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345468+0000", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4508e-05 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:8 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:12.236 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.906672138000005, 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019792753, 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325868+0000", 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325866+0000", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325872+0000", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1139999999999997e-06 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325884+0000", 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2500000000000001e-05 2024-04-24T16:01:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325890+0000", 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7830000000000004e-06 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331282+0000", 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005391725 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:12.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331368+0000", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6907e-05 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345510+0000", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014141733 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345583+0000", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3288999999999996e-05 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345661+0000", 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7368999999999997e-05 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(osd.4.11:9 1.7 1:e00fa4b3:::rbd_data.129ad8b6ae3f.0000000000000002:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.906640769999996, 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026341205999999999, 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325899+0000", 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:12.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325903+0000", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1459999999999998e-06 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325912+0000", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6670000000000002e-06 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.325916+0000", 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3680000000000004e-06 2024-04-24T16:01:12.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331398+0000", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054817959999999997 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.331461+0000", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.355e-05 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345687+0000", 2024-04-24T16:01:12.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014225326999999999 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.345756+0000", 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8738999999999994e-05 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.352241+0000", 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064851010000000001 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.245 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:43 1.3 e44/12 1:cf9d03c9:::rbd_data.129ad8b6ae3f.0000000000000005:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.882917043999996, 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044985665000000001, 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349623+0000", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.349622+0000", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377435+0000", 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027813747999999999 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377458+0000", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377469+0000", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0383e-05 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377517+0000", 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7923000000000002e-05 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377543+0000", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6044999999999999e-05 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394532+0000", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016988965000000002 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.394609+0000", 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7049e-05 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:26 3.2 e44/15 3:54cf260f:::rbd_data.129ad8b6ae3f.0000000000000000:head v 44'6, mlcod=43'4)", 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.855059306000001, 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037118194, 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377481+0000", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.377480+0000", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382066+0000", 2024-04-24T16:01:12.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045855959999999999 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382082+0000", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5821e-05 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382089+0000", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4800000000000004e-06 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382136+0000", 2024-04-24T16:01:12.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7101999999999997e-05 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382158+0000", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1648e-05 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414526+0000", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032367901999999997 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.414599+0000", 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3355000000000005e-05 2024-04-24T16:01:12.253 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:45 1.3 e44/12 1:dc47b767:::rbd_data.129ad8b6ae3f.0000000000000006:head v 44'2, mlcod=0'0)", 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.850429801000004, 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039710431999999997, 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.382110+0000", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388877+0000", 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067672660000000001 2024-04-24T16:01:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388895+0000", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8077999999999999e-05 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388904+0000", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5229999999999995e-06 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388943+0000", 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9134999999999997e-05 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388960+0000", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7455000000000001e-05 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421782+0000", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032822005000000001 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421821+0000", 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.257 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:29 3.2 e44/15 3:65064853:::rbd_data.129ad8b6ae3f.0000000000000001:head v 44'7, mlcod=43'4)", 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.843622257999996, 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033088908, 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388918+0000", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.388917+0000", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396220+0000", 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073029899999999997 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396241+0000", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1580999999999999e-05 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396250+0000", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7900000000000005e-06 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396296+0000", 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5478000000000001e-05 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.396316+0000", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0514999999999999e-05 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.421969+0000", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025652531999999999 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.422007+0000", 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8158000000000003e-05 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:36 3.2 e44/15 3:4d625aa2:::rbd_data.129ad8b6ae3f.0000000000000007:head v 44'8, mlcod=44'5)", 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.813082030999993, 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099034450000000003, 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419458+0000", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.419457+0000", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423339+0000", 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038817869999999998 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423350+0000", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1195e-05 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423356+0000", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4350000000000002e-06 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423402+0000", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5623e-05 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.423419+0000", 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581999999999999e-05 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429308+0000", 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058888430000000004 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.429362+0000", 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3440000000000003e-05 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:41 3.2 e44/15 3:473eece9:::rbd_data.129ad8b6ae3f.0000000000000008:head v 44'9, mlcod=44'7)", 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.797442351000001, 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027081411999999999, 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435098+0000", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.435097+0000", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443743+0000", 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086461239999999998 2024-04-24T16:01:12.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443768+0000", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443778+0000", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443833+0000", 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5537e-05 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.443859+0000", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5507000000000002e-05 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462140+0000", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018281677 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.462179+0000", 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825000000000002e-05 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.269 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4762.0:64 1.7 1:ed2f5aab:::rbd_data.129ad8b6ae3f.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.758267102000005, 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010529643, 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474273+0000", 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.474271+0000", 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475072+0000", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000801161 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475094+0000", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1582000000000001e-05 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475103+0000", 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4609999999999997e-06 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475175+0000", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099000000000002e-05 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475279+0000", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330199999999999 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.475380+0000", 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101751 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484364+0000", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089838629999999999 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484724+0000", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036010099999999999 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484766+0000", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1919999999999998e-05 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.484803+0000", 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6579000000000003e-05 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4762.0:63 1.5 e44/34 1:b77d1426:::rbd_data.129ad8b6ae3f.000000000000000e:head v 44'1, mlcod=0'0)", 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.745648982000006, 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02073351, 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486891+0000", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.486890+0000", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488280+0000", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389892 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488301+0000", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1175999999999999e-05 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488310+0000", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5490000000000006e-06 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488355+0000", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5524e-05 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.488381+0000", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5965e-05 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507580+0000", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019199091000000001 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.507625+0000", 2024-04-24T16:01:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4382999999999998e-05 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:32 1.3 e44/12 1:c97825ec:ns::rbd_data.1308dd266c0.0000000000000000:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.623252844999996, 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018219637, 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609287+0000", 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.609286+0000", 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615763+0000", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064772950000000001 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615783+0000", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0017000000000002e-05 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615793+0000", 2024-04-24T16:01:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7089999999999997e-06 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615844+0000", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088999999999998e-05 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.615863+0000", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9480999999999999e-05 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627444+0000", 2024-04-24T16:01:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581088999999999 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.627507+0000", 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2595999999999995e-05 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:40 1.3 e44/12 1:cd82dd04:ns::rbd_data.1308dd266c0.0000000000000005:head v 44'5, mlcod=44'2)", 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.609723637000002, 2024-04-24T16:01:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030241023999999998, 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622817+0000", 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.622816+0000", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629187+0000", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063713019999999997 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629199+0000", 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1601e-05 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629206+0000", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.486e-06 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629238+0000", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1664e-05 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.629271+0000", 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3668999999999999e-05 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653003+0000", 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023731702 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.653058+0000", 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4567000000000001e-05 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:42 1.7 1:f3bf9fb2:ns::rbd_data.1308dd266c0.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.598989309000004, 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030834810000000001, 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633551+0000", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.633548+0000", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999971 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635112+0000", 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015643300000000001 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635135+0000", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2886000000000001e-05 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635147+0000", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2026e-05 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635222+0000", 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.483e-05 2024-04-24T16:01:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635343+0000", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120499 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.635447+0000", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104072 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.662027+0000", 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026579985 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664273+0000", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022466320000000001 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664330+0000", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6925999999999997e-05 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664386+0000", 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5581999999999997e-05 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:45 1.7 1:e1a28c57:ns::rbd_data.1308dd266c0.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.584693165999994, 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016919120999999999, 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647847+0000", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.647845+0000", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648717+0000", 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000872561 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648735+0000", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7419999999999999e-05 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648741+0000", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9499999999999998e-06 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648783+0000", 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2423999999999999e-05 2024-04-24T16:01:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648888+0000", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010509 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.648991+0000", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010278899999999999 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664602+0000", 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015610929000000001 2024-04-24T16:01:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664670+0000", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8476e-05 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664728+0000", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7249e-05 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.664766+0000", 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8649999999999998e-05 2024-04-24T16:01:12.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:56 1.7 1:f2073c35:ns::rbd_data.1308dd266c0.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.552466073999994, 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034381079000000002, 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:12.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680074+0000", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680072+0000", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.680995+0000", 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092334600000000002 2024-04-24T16:01:12.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681016+0000", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0988999999999998e-05 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681025+0000", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3100000000000001e-06 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681086+0000", 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0770000000000003e-05 2024-04-24T16:01:12.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681236+0000", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000150583 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.681402+0000", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016575699999999999 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.705828+0000", 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024425941999999999 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714368+0000", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085397570000000003 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714405+0000", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7672000000000001e-05 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714455+0000", 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9956000000000001e-05 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.4872.0:58 1.7 1:e6b25fcf:ns::rbd_data.1308dd266c0.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.546370233000005, 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029857888999999999, 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686170+0000", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.686168+0000", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687185+0000", 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001016905 2024-04-24T16:01:12.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687205+0000", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.003e-05 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687215+0000", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7890000000000006e-06 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687275+0000", 2024-04-24T16:01:12.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9852999999999998e-05 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687411+0000", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013622600000000001 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.687547+0000", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013615599999999999 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714135+0000", 2024-04-24T16:01:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026587547999999999 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715943+0000", 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00180777 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.715992+0000", 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9660999999999998e-05 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.716028+0000", 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5612999999999999e-05 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:57 1.5 e44/34 1:ba8a8c25:ns::rbd_data.1308dd266c0.000000000000000d:head v 44'3, mlcod=44'1)", 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.542915676000007, 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024707577000000001, 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689625+0000", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.689623+0000", 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691705+0000", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020818659999999999 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691723+0000", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7674999999999999e-05 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691732+0000", 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9290000000000004e-06 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691777+0000", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5009999999999998e-05 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.691803+0000", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5927000000000001e-05 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714289+0000", 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022485987999999998 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.714332+0000", 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3402000000000002e-05 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.4872.0:60 1.5 e44/34 1:bc09ac1f:ns::rbd_data.1308dd266c0.000000000000000f:head v 44'4, mlcod=44'2)", 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.507471554999995, 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011581014000000001, 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.312 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725069+0000", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725067+0000", 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725896+0000", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082877199999999997 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725915+0000", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8700000000000001e-05 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725925+0000", 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0513000000000001e-05 2024-04-24T16:01:12.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725970+0000", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4286999999999998e-05 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.725992+0000", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2543000000000001e-05 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736609+0000", 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010616947 2024-04-24T16:01:12.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:48.736650+0000", 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0547000000000003e-05 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.317 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(osd.1.0:165 3.2 e50/15 3:4136e5c4:::rbd_data.129ad8b6ae3f.000000000000000d:head v 50'64, mlcod=50'62)", 2024-04-24T16:01:12.317 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 76.456011790000005, 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010817801, 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776528+0000", 2024-04-24T16:01:12.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.776527+0000", 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780299+0000", 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037720800000000001 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-04-24T16:01:12.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780316+0000", 2024-04-24T16:01:12.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6307000000000002e-05 2024-04-24T16:01:12.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780324+0000", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780363+0000", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8986999999999998e-05 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.780384+0000", 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0931e-05 2024-04-24T16:01:12.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787311+0000", 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069266270000000003 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-04-24T15:59:55.787346+0000", 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5568999999999998e-05 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-04-24T16:01:12.324 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-24T16:01:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-04-24T16:01:16.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-24T16:01:17.126 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a: {} 2024-04-24T16:01:17.137 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b: {} 2024-04-24T16:01:17.147 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c: {} 2024-04-24T16:01:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T16:01:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T16:01:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T16:01:17.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T16:01:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T16:01:18.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-24T16:01:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-24T16:01:19.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T16:01:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T16:01:20.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T16:01:21.093 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T16:01:21.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T16:01:22.099 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T16:01:22.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-24T16:01:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-24T16:01:23.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T16:01:24.112 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T16:01:24.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-24T16:01:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-24T16:01:25.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T16:01:26.125 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T16:01:26.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-24T16:01:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-24T16:01:27.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T16:01:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T16:01:28.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-24T16:01:29.141 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-24T16:01:29.155 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-24T16:01:29.155 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-24T16:01:29.155 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-24T16:01:29.155 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-24T16:01:29.156 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T16:01:29.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T16:01:29.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:29.510 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":114,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T16:01:29.135327+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T15:58:48.277449+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:01:29.526 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T16:01:29.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T16:01:29.876 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:29.876 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":114,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T16:01:29.135327+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T15:58:48.277449+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:01:29.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T16:01:29.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T16:01:29.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T16:01:29.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T16:01:29.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T16:01:29.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T16:01:30.125 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:30.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T16:01:30.128 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:30.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T16:01:30.170 INFO:teuthology.orchestra.run.smithi032.stdout:47244640293 2024-04-24T16:01:30.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T16:01:30.174 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:30.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T16:01:30.181 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:30.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T16:01:30.190 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:30.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T16:01:30.599 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:30.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640294 for osd.3 2024-04-24T16:01:30.623 INFO:teuthology.orchestra.run.smithi032.stdout:47244640292 2024-04-24T16:01:30.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640292 for osd.5 2024-04-24T16:01:30.750 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:30.758 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:30.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640294 for osd.0 2024-04-24T16:01:30.773 INFO:teuthology.orchestra.run.smithi032.stdout:47244640293 2024-04-24T16:01:30.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640294 for osd.1 2024-04-24T16:01:30.786 INFO:teuthology.orchestra.run.smithi032.stdout:47244640293 2024-04-24T16:01:30.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640294 got 47244640293 for osd.4 2024-04-24T16:01:30.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640294 got 47244640293 for osd.2 2024-04-24T16:01:31.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T16:01:31.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T16:01:31.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T16:01:31.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T16:01:31.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T16:01:31.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T16:01:32.064 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:32.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640294 for osd.3 2024-04-24T16:01:32.136 INFO:teuthology.orchestra.run.smithi032.stdout:47244640292 2024-04-24T16:01:32.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640292 for osd.5 2024-04-24T16:01:32.213 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:32.234 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640295 for osd.0 2024-04-24T16:01:32.234 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:32.259 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:32.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640294 got 47244640294 for osd.4 2024-04-24T16:01:32.275 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:32.311 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:32.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640295 for osd.1 2024-04-24T16:01:32.329 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:32.355 INFO:teuthology.orchestra.run.smithi032.stdout:47244640294 2024-04-24T16:01:32.371 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640294 got 47244640294 for osd.2 2024-04-24T16:01:32.371 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:33.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T16:01:33.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T16:01:33.508 INFO:teuthology.orchestra.run.smithi032.stdout:47244640295 2024-04-24T16:01:33.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640295 for osd.3 2024-04-24T16:01:33.549 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:33.557 INFO:teuthology.orchestra.run.smithi032.stdout:47244640293 2024-04-24T16:01:33.573 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640293 for osd.5 2024-04-24T16:01:33.573 DEBUG:teuthology.parallel:result is None 2024-04-24T16:01:33.573 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T16:01:33.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:33.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:33.979 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-24T16:01:33.783104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.655579"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.624}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.437}]}]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.501}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:33.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:34.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:34.332 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:34.345 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-24T16:01:33.783104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.655579"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.624}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.437}]}]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.501}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:34.346 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T16:01:34.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-24T16:01:34.716 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T16:01:34.726 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-04-24T16:01:34.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T16:01:35.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:35.079 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":114,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T16:01:29.135327+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T15:58:48.277449+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:01:35.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T16:01:35.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:35.445 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:35.447 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:35.460 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-24T16:01:33.783104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.655579"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.624}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.437}]}]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.501}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:35.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:35.814 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:35.827 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-24T16:01:35.783583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001827"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.624}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.437}]}]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.501}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:35.829 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T16:01:35.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:36.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:36.192 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-24T16:01:35.783583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001827"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.373},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.4199999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.624}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.437}]}]},{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48699999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.432}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.501}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:36.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T16:01:36.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:36.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":114,"fsid":"c4fc9996-5ff9-4e71-b213-9020967fae1a","created":"2024-04-24T15:58:30.292182+0000","modified":"2024-04-24T16:01:29.135327+0000","last_up_change":"2024-04-24T15:58:41.011852+0000","last_in_change":"2024-04-24T15:58:32.987203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T15:58:41.444445+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T15:58:41.808270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T15:58:48.277449+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":1,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"51c39890-e79b-4030-a4a0-8c98ec2018ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":41951},{"type":"v1","addr":"172.21.15.32:6809","nonce":41951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":41951},{"type":"v1","addr":"172.21.15.32:6811","nonce":41951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":41951},{"type":"v1","addr":"172.21.15.32:6819","nonce":41951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6813","nonce":41951},{"type":"v1","addr":"172.21.15.32:6815","nonce":41951}]},"public_addr":"172.21.15.32:6809/41951","cluster_addr":"172.21.15.32:6811/41951","heartbeat_back_addr":"172.21.15.32:6819/41951","heartbeat_front_addr":"172.21.15.32:6815/41951","state":["exists","up"]},{"osd":1,"uuid":"1fb2523f-db36-46d9-80c4-dc08ac530daa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":41952},{"type":"v1","addr":"172.21.15.32:6801","nonce":41952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":41952},{"type":"v1","addr":"172.21.15.32:6803","nonce":41952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":41952},{"type":"v1","addr":"172.21.15.32:6807","nonce":41952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":41952},{"type":"v1","addr":"172.21.15.32:6805","nonce":41952}]},"public_addr":"172.21.15.32:6801/41952","cluster_addr":"172.21.15.32:6803/41952","heartbeat_back_addr":"172.21.15.32:6807/41952","heartbeat_front_addr":"172.21.15.32:6805/41952","state":["exists","up"]},{"osd":2,"uuid":"7d4dd13f-26d8-4c39-a6a6-020e91634f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":41953},{"type":"v1","addr":"172.21.15.32:6814","nonce":41953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":41953},{"type":"v1","addr":"172.21.15.32:6818","nonce":41953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":41953},{"type":"v1","addr":"172.21.15.32:6823","nonce":41953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":41953},{"type":"v1","addr":"172.21.15.32:6821","nonce":41953}]},"public_addr":"172.21.15.32:6814/41953","cluster_addr":"172.21.15.32:6818/41953","heartbeat_back_addr":"172.21.15.32:6823/41953","heartbeat_front_addr":"172.21.15.32:6821/41953","state":["exists","up"]},{"osd":3,"uuid":"03ed1eb6-8fa9-497c-b769-2743acc0e6aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":41923},{"type":"v1","addr":"172.21.15.86:6803","nonce":41923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":41923},{"type":"v1","addr":"172.21.15.86:6807","nonce":41923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":41923},{"type":"v1","addr":"172.21.15.86:6815","nonce":41923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":41923},{"type":"v1","addr":"172.21.15.86:6811","nonce":41923}]},"public_addr":"172.21.15.86:6803/41923","cluster_addr":"172.21.15.86:6807/41923","heartbeat_back_addr":"172.21.15.86:6815/41923","heartbeat_front_addr":"172.21.15.86:6811/41923","state":["exists","up"]},{"osd":4,"uuid":"f7ff11ec-62d6-44f3-bea1-7ff62d280e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41924},{"type":"v1","addr":"172.21.15.86:6817","nonce":41924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41924},{"type":"v1","addr":"172.21.15.86:6819","nonce":41924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41924},{"type":"v1","addr":"172.21.15.86:6823","nonce":41924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41924},{"type":"v1","addr":"172.21.15.86:6821","nonce":41924}]},"public_addr":"172.21.15.86:6817/41924","cluster_addr":"172.21.15.86:6819/41924","heartbeat_back_addr":"172.21.15.86:6823/41924","heartbeat_front_addr":"172.21.15.86:6821/41924","state":["exists","up"]},{"osd":5,"uuid":"423ca5ac-f6a3-496e-a4c5-9ac205e2e474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41925},{"type":"v1","addr":"172.21.15.86:6802","nonce":41925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41925},{"type":"v1","addr":"172.21.15.86:6806","nonce":41925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41925},{"type":"v1","addr":"172.21.15.86:6814","nonce":41925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41925},{"type":"v1","addr":"172.21.15.86:6810","nonce":41925}]},"public_addr":"172.21.15.86:6802/41925","cluster_addr":"172.21.15.86:6806/41925","heartbeat_back_addr":"172.21.15.86:6814/41925","heartbeat_front_addr":"172.21.15.86:6810/41925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.171683+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.085846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.118237+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.172469+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.139953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:58:38.084518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:01:37.562 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T16:01:37.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:37.728 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:37.728 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:37.728 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:37.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T16:01:38.094 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T16:01:38.106 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T16:01:38.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:38.263 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:38.264 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:38.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T16:01:38.626 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T16:01:38.639 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T16:01:38.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:38.795 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:38.796 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:38.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T16:01:39.164 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T16:01:39.178 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T16:01:39.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:39.338 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:39.338 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:39.338 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:39.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T16:01:39.706 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T16:01:39.718 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T16:01:39.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:39.877 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:39.878 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:39.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T16:01:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T16:01:40.263 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T16:01:40.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T16:01:40.421 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T16:01:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T16:01:40.421 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T16:01:40.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T16:01:40.787 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T16:01:40.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:01:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:01:41.150 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:01:41.162 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-24T16:01:39.784513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977752,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253288,"statfs":{"total":579820584960,"available":577795366912,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.646675"},"pg_stats":[{"pgid":"1.5","version":"96'8","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260360+0000","last_change":"2024-04-24T16:01:08.928968+0000","last_active":"2024-04-24T16:01:28.260360+0000","last_peered":"2024-04-24T16:01:28.260360+0000","last_clean":"2024-04-24T16:01:28.260360+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:28.260360+0000","last_undegraded":"2024-04-24T16:01:28.260360+0000","last_fullsized":"2024-04-24T16:01:28.260360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:41.013126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"96'11","reported_seq":222,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133542+0000","last_change":"2024-04-24T16:01:08.925683+0000","last_active":"2024-04-24T16:01:28.133542+0000","last_peered":"2024-04-24T16:01:28.133542+0000","last_clean":"2024-04-24T16:01:28.133542+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:28.133542+0000","last_undegraded":"2024-04-24T16:01:28.133542+0000","last_fullsized":"2024-04-24T16:01:28.133542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:41.469592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"96'7","reported_seq":162,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133427+0000","last_change":"2024-04-24T16:01:08.924989+0000","last_active":"2024-04-24T16:01:28.133427+0000","last_peered":"2024-04-24T16:01:28.133427+0000","last_clean":"2024-04-24T16:01:28.133427+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:28.133427+0000","last_undegraded":"2024-04-24T16:01:28.133427+0000","last_fullsized":"2024-04-24T16:01:28.133427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:25.932031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'117","reported_seq":280,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260446+0000","last_change":"2024-04-24T16:00:41.451428+0000","last_active":"2024-04-24T16:01:28.260446+0000","last_peered":"2024-04-24T16:01:28.260446+0000","last_clean":"2024-04-24T16:01:28.260446+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:28.260446+0000","last_undegraded":"2024-04-24T16:01:28.260446+0000","last_fullsized":"2024-04-24T16:01:28.260446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:41.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"96'139","reported_seq":661,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133824+0000","last_change":"2024-04-24T16:01:08.925894+0000","last_active":"2024-04-24T16:01:28.133824+0000","last_peered":"2024-04-24T16:01:28.133824+0000","last_clean":"2024-04-24T16:01:28.133824+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:28.133824+0000","last_undegraded":"2024-04-24T16:01:28.133824+0000","last_fullsized":"2024-04-24T16:01:28.133824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:50.031327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"96'10","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260350+0000","last_change":"2024-04-24T16:01:08.928785+0000","last_active":"2024-04-24T16:01:28.260350+0000","last_peered":"2024-04-24T16:01:28.260350+0000","last_clean":"2024-04-24T16:01:28.260350+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:28.260350+0000","last_undegraded":"2024-04-24T16:01:28.260350+0000","last_fullsized":"2024-04-24T16:01:28.260350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:09:10.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"96'530","reported_seq":1987,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133428+0000","last_change":"2024-04-24T16:01:08.925220+0000","last_active":"2024-04-24T16:01:28.133428+0000","last_peered":"2024-04-24T16:01:28.133428+0000","last_clean":"2024-04-24T16:01:28.133428+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:28.133428+0000","last_undegraded":"2024-04-24T16:01:28.133428+0000","last_fullsized":"2024-04-24T16:01:28.133428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:25.049259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"96'10","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260477+0000","last_change":"2024-04-24T16:01:09.019084+0000","last_active":"2024-04-24T16:01:28.260477+0000","last_peered":"2024-04-24T16:01:28.260477+0000","last_clean":"2024-04-24T16:01:28.260477+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:28.260477+0000","last_undegraded":"2024-04-24T16:01:28.260477+0000","last_fullsized":"2024-04-24T16:01:28.260477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:40:42.986056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'278","reported_seq":1308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133442+0000","last_change":"2024-04-24T16:01:08.926052+0000","last_active":"2024-04-24T16:01:28.133442+0000","last_peered":"2024-04-24T16:01:28.133442+0000","last_clean":"2024-04-24T16:01:28.133442+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:28.133442+0000","last_undegraded":"2024-04-24T16:01:28.133442+0000","last_fullsized":"2024-04-24T16:01:28.133442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:01.880873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133946+0000","last_change":"2024-04-24T16:01:08.990280+0000","last_active":"2024-04-24T16:01:28.133946+0000","last_peered":"2024-04-24T16:01:28.133946+0000","last_clean":"2024-04-24T16:01:28.133946+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:28.133946+0000","last_undegraded":"2024-04-24T16:01:28.133946+0000","last_fullsized":"2024-04-24T16:01:28.133946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:41.064660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"96'183","reported_seq":1447,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260364+0000","last_change":"2024-04-24T16:01:08.929604+0000","last_active":"2024-04-24T16:01:28.260364+0000","last_peered":"2024-04-24T16:01:28.260364+0000","last_clean":"2024-04-24T16:01:28.260364+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:28.260364+0000","last_undegraded":"2024-04-24T16:01:28.260364+0000","last_fullsized":"2024-04-24T16:01:28.260364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:49.047185+0000","last_clean_scrub_stamp":"2024-04-24T15:58:49.047185+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:25:49.582079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'8","reported_seq":72,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.260504+0000","last_change":"2024-04-24T16:01:09.018890+0000","last_active":"2024-04-24T16:01:28.260504+0000","last_peered":"2024-04-24T16:01:28.260504+0000","last_clean":"2024-04-24T16:01:28.260504+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:28.260504+0000","last_undegraded":"2024-04-24T16:01:28.260504+0000","last_fullsized":"2024-04-24T16:01:28.260504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:12.393610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"96'13","reported_seq":196,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T16:01:28.133978+0000","last_change":"2024-04-24T16:01:08.990968+0000","last_active":"2024-04-24T16:01:28.133978+0000","last_peered":"2024-04-24T16:01:28.133978+0000","last_clean":"2024-04-24T16:01:28.133978+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:28.133978+0000","last_undegraded":"2024-04-24T16:01:28.133978+0000","last_fullsized":"2024-04-24T16:01:28.133978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:58:42.017474+0000","last_clean_scrub_stamp":"2024-04-24T15:58:42.017474+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:43.427991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1130,"ondisk_log_size":1130,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323288,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048552,"statfs":{"total":96636764160,"available":96305717248,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48399999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.498},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.36799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.623,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.378},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.377},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.438}]},{"osd":4,"last update":"Wed Apr 24 16:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.622},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.52100000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335564,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036276,"statfs":{"total":96636764160,"available":96293146624,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70099999999999996,"5min":0.747,"15min":0.747},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.42299999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.621,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.51400000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.503,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.45700000000000002,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56200000000000006,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.59999999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327976,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043864,"statfs":{"total":96636764160,"available":96300916736,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41499999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.496},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60599999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.621,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.434},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53000000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.56799999999999995,"5min":0.629,"15min":0.629},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.50900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72799999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65100000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69699999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640297,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335580,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036260,"statfs":{"total":96636764160,"available":96293130240,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.504,"15min":0.504},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36299999999999999,"15min":0.36399999999999999},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.317},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.371}]},{"osd":3,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.46999999999999997}]},{"osd":5,"last update":"Wed Apr 24 16:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.747,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.42099999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331476,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040364,"statfs":{"total":96636764160,"available":96297332736,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36599999999999999,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55700000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.375},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.505,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.34200000000000003}]},{"osd":3,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.70999999999999996}]},{"osd":5,"last update":"Wed Apr 24 16:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-24T15:58:49.047185+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.167 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-24T15:58:49.047185+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-24T15:58:49.047185+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-24T15:58:49.047185+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-24T15:58:42.017474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=58, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=16, tm_min=1, tm_sec=36, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T16:01:41.168 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T16:02:01.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T16:02:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T16:02:01.533 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-24T16:02:01.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-04-24T16:01:59.788529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84477060,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":46,"num_read":7822,"num_read_kb":178446,"num_write":1161,"num_write_kb":273546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":130614189,"num_keys_recovered":36,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":268,"num_omap_keys":28,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1403,"ondisk_log_size":1403,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1977712,"kb_used_data":173120,"kb_used_omap":9,"kb_used_meta":1804534,"kb_avail":564253328,"statfs":{"total":579820584960,"available":577795407872,"internally_reserved":0,"allocated":177274880,"data_stored":172388202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1847842832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43,"num_omap_keys":8,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002480"},"pg_stats":[{"pgid":"1.5","version":"114'14","reported_seq":197,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:52.987711+0000","last_change":"2024-04-24T16:01:52.987638+0000","last_active":"2024-04-24T16:01:52.987711+0000","last_peered":"2024-04-24T16:01:52.987711+0000","last_clean":"2024-04-24T16:01:52.987711+0000","last_became_active":"2024-04-24T16:00:22.364098+0000","last_became_peered":"2024-04-24T16:00:22.364098+0000","last_unstale":"2024-04-24T16:01:52.987711+0000","last_undegraded":"2024-04-24T16:01:52.987711+0000","last_fullsized":"2024-04-24T16:01:52.987711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-04-24T16:01:52.987547+0000","last_deep_scrub":"114'14","last_deep_scrub_stamp":"2024-04-24T16:01:52.987547+0000","last_clean_scrub_stamp":"2024-04-24T16:01:52.987547+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:26:36.133381+0000","scrub_duration":0.022571905999999999,"objects_trimmed":0,"snaptrim_duration":0.000385215,"stat_sum":{"num_bytes":12583028,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":12,"num_write_kb":12297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42,"num_omap_keys":7,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"114'18","reported_seq":240,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:40.151930+0000","last_change":"2024-04-24T16:01:40.151889+0000","last_active":"2024-04-24T16:01:40.151930+0000","last_peered":"2024-04-24T16:01:40.151930+0000","last_clean":"2024-04-24T16:01:40.151930+0000","last_became_active":"2024-04-24T16:00:57.876040+0000","last_became_peered":"2024-04-24T16:00:57.876040+0000","last_unstale":"2024-04-24T16:01:40.151930+0000","last_undegraded":"2024-04-24T16:01:40.151930+0000","last_fullsized":"2024-04-24T16:01:40.151930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'18","last_scrub_stamp":"2024-04-24T16:01:40.151807+0000","last_deep_scrub":"114'18","last_deep_scrub_stamp":"2024-04-24T16:01:40.151807+0000","last_clean_scrub_stamp":"2024-04-24T16:01:40.151807+0000","objects_scrubbed":7,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.200720+0000","scrub_duration":0.026984382000000001,"objects_trimmed":0,"snaptrim_duration":0.00042044899999999999,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":40,"num_write":11,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50331648,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":109,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"114'12","reported_seq":178,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:42.222346+0000","last_change":"2024-04-24T16:01:42.222316+0000","last_active":"2024-04-24T16:01:42.222346+0000","last_peered":"2024-04-24T16:01:42.222346+0000","last_clean":"2024-04-24T16:01:42.222346+0000","last_became_active":"2024-04-24T15:58:43.033259+0000","last_became_peered":"2024-04-24T15:58:43.033259+0000","last_unstale":"2024-04-24T16:01:42.222346+0000","last_undegraded":"2024-04-24T16:01:42.222346+0000","last_fullsized":"2024-04-24T16:01:42.222346+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'12","last_scrub_stamp":"2024-04-24T16:01:42.222188+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-04-24T16:01:42.222188+0000","last_clean_scrub_stamp":"2024-04-24T16:01:42.222188+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:25:14.216684+0000","scrub_duration":0.023922259000000001,"objects_trimmed":0,"snaptrim_duration":0.000473234,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26,"num_write":7,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":73,"num_omap_keys":8,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"114'119","reported_seq":293,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:48.735606+0000","last_change":"2024-04-24T16:01:48.735578+0000","last_active":"2024-04-24T16:01:48.735606+0000","last_peered":"2024-04-24T16:01:48.735606+0000","last_clean":"2024-04-24T16:01:48.735606+0000","last_became_active":"2024-04-24T16:00:41.451039+0000","last_became_peered":"2024-04-24T16:00:41.451039+0000","last_unstale":"2024-04-24T16:01:48.735606+0000","last_undegraded":"2024-04-24T16:01:48.735606+0000","last_fullsized":"2024-04-24T16:01:48.735606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'119","last_scrub_stamp":"2024-04-24T16:01:48.735484+0000","last_deep_scrub":"114'119","last_deep_scrub_stamp":"2024-04-24T16:01:48.735484+0000","last_clean_scrub_stamp":"2024-04-24T16:01:48.735484+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:12:03.020433+0000","scrub_duration":0.0047509709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"114'143","reported_seq":676,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:38.994318+0000","last_change":"2024-04-24T16:01:38.994254+0000","last_active":"2024-04-24T16:01:38.994318+0000","last_peered":"2024-04-24T16:01:38.994318+0000","last_clean":"2024-04-24T16:01:38.994318+0000","last_became_active":"2024-04-24T15:58:50.065979+0000","last_became_peered":"2024-04-24T15:58:50.065979+0000","last_unstale":"2024-04-24T16:01:38.994318+0000","last_undegraded":"2024-04-24T16:01:38.994318+0000","last_fullsized":"2024-04-24T16:01:38.994318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'143","last_scrub_stamp":"2024-04-24T16:01:38.994094+0000","last_deep_scrub":"114'143","last_deep_scrub_stamp":"2024-04-24T16:01:38.994094+0000","last_clean_scrub_stamp":"2024-04-24T16:01:38.994094+0000","objects_scrubbed":7,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:04:28.355326+0000","scrub_duration":0.0055803190000000003,"objects_trimmed":0,"snaptrim_duration":0.00039322599999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":7,"num_read":621,"num_read_kb":11195,"num_write":123,"num_write_kb":8525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"114'15","reported_seq":206,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:49.009523+0000","last_change":"2024-04-24T16:01:49.009498+0000","last_active":"2024-04-24T16:01:49.009523+0000","last_peered":"2024-04-24T16:01:49.009523+0000","last_clean":"2024-04-24T16:01:49.009523+0000","last_became_active":"2024-04-24T15:58:43.032805+0000","last_became_peered":"2024-04-24T15:58:43.032805+0000","last_unstale":"2024-04-24T16:01:49.009523+0000","last_undegraded":"2024-04-24T16:01:49.009523+0000","last_fullsized":"2024-04-24T16:01:49.009523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'15","last_scrub_stamp":"2024-04-24T16:01:49.009412+0000","last_deep_scrub":"114'15","last_deep_scrub_stamp":"2024-04-24T16:01:49.009412+0000","last_clean_scrub_stamp":"2024-04-24T16:01:49.009412+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:39:05.642743+0000","scrub_duration":0.033749517,"objects_trimmed":0,"snaptrim_duration":0.000257233,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":4736,"num_write":10,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"114'544","reported_seq":2012,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:48.163052+0000","last_change":"2024-04-24T16:01:48.162987+0000","last_active":"2024-04-24T16:01:48.163052+0000","last_peered":"2024-04-24T16:01:48.163052+0000","last_clean":"2024-04-24T16:01:48.163052+0000","last_became_active":"2024-04-24T16:00:25.380750+0000","last_became_peered":"2024-04-24T16:00:25.380750+0000","last_unstale":"2024-04-24T16:01:48.163052+0000","last_undegraded":"2024-04-24T16:01:48.163052+0000","last_fullsized":"2024-04-24T16:01:48.163052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'544","last_scrub_stamp":"2024-04-24T16:01:48.162857+0000","last_deep_scrub":"114'544","last_deep_scrub_stamp":"2024-04-24T16:01:48.162857+0000","last_clean_scrub_stamp":"2024-04-24T16:01:48.162857+0000","objects_scrubbed":14,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:56:17.290629+0000","scrub_duration":0.012206844999999999,"objects_trimmed":0,"snaptrim_duration":0.00037344899999999998,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":4535,"num_read_kb":48942,"num_write":477,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"114'15","reported_seq":169,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:43.701940+0000","last_change":"2024-04-24T16:01:43.701913+0000","last_active":"2024-04-24T16:01:43.701940+0000","last_peered":"2024-04-24T16:01:43.701940+0000","last_clean":"2024-04-24T16:01:43.701940+0000","last_became_active":"2024-04-24T15:58:43.031347+0000","last_became_peered":"2024-04-24T15:58:43.031347+0000","last_unstale":"2024-04-24T16:01:43.701940+0000","last_undegraded":"2024-04-24T16:01:43.701940+0000","last_fullsized":"2024-04-24T16:01:43.701940+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'15","last_scrub_stamp":"2024-04-24T16:01:43.701821+0000","last_deep_scrub":"114'15","last_deep_scrub_stamp":"2024-04-24T16:01:43.701821+0000","last_clean_scrub_stamp":"2024-04-24T16:01:43.701821+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:13:00.897816+0000","scrub_duration":0.017687339,"objects_trimmed":0,"snaptrim_duration":0.0010629890000000001,"stat_sum":{"num_bytes":8388724,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":9,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"114'292","reported_seq":1333,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:50.211295+0000","last_change":"2024-04-24T16:01:50.211257+0000","last_active":"2024-04-24T16:01:50.211295+0000","last_peered":"2024-04-24T16:01:50.211295+0000","last_clean":"2024-04-24T16:01:50.211295+0000","last_became_active":"2024-04-24T15:58:50.087739+0000","last_became_peered":"2024-04-24T15:58:50.087739+0000","last_unstale":"2024-04-24T16:01:50.211295+0000","last_undegraded":"2024-04-24T16:01:50.211295+0000","last_fullsized":"2024-04-24T16:01:50.211295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'292","last_scrub_stamp":"2024-04-24T16:01:50.211146+0000","last_deep_scrub":"114'292","last_deep_scrub_stamp":"2024-04-24T16:01:50.211146+0000","last_clean_scrub_stamp":"2024-04-24T16:01:50.211146+0000","objects_scrubbed":14,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:37:20.451956+0000","scrub_duration":0.013203629,"objects_trimmed":0,"snaptrim_duration":0.0010855699999999999,"stat_sum":{"num_bytes":19,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":13,"num_read":932,"num_read_kb":64633,"num_write":216,"num_write_kb":56008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"96'1","reported_seq":145,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:41.707619+0000","last_change":"2024-04-24T16:01:41.707566+0000","last_active":"2024-04-24T16:01:41.707619+0000","last_peered":"2024-04-24T16:01:41.707619+0000","last_clean":"2024-04-24T16:01:41.707619+0000","last_became_active":"2024-04-24T15:58:43.033391+0000","last_became_peered":"2024-04-24T15:58:43.033391+0000","last_unstale":"2024-04-24T16:01:41.707619+0000","last_undegraded":"2024-04-24T16:01:41.707619+0000","last_fullsized":"2024-04-24T16:01:41.707619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'1","last_scrub_stamp":"2024-04-24T16:01:41.707453+0000","last_deep_scrub":"96'1","last_deep_scrub_stamp":"2024-04-24T16:01:41.707453+0000","last_clean_scrub_stamp":"2024-04-24T16:01:41.707453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:34.137987+0000","scrub_duration":0.002313117,"objects_trimmed":0,"snaptrim_duration":0.00030557799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"114'198","reported_seq":1473,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:54.980657+0000","last_change":"2024-04-24T16:01:54.980630+0000","last_active":"2024-04-24T16:01:54.980657+0000","last_peered":"2024-04-24T16:01:54.980657+0000","last_clean":"2024-04-24T16:01:54.980657+0000","last_became_active":"2024-04-24T15:58:50.065792+0000","last_became_peered":"2024-04-24T15:58:50.065792+0000","last_unstale":"2024-04-24T16:01:54.980657+0000","last_undegraded":"2024-04-24T16:01:54.980657+0000","last_fullsized":"2024-04-24T16:01:54.980657+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'198","last_scrub_stamp":"2024-04-24T16:01:54.980536+0000","last_deep_scrub":"114'198","last_deep_scrub_stamp":"2024-04-24T16:01:54.980536+0000","last_clean_scrub_stamp":"2024-04-24T16:01:54.980536+0000","objects_scrubbed":15,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:54.536944+0000","scrub_duration":0.010348293999999999,"objects_trimmed":0,"snaptrim_duration":0.00091556000000000005,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":14,"num_read":1493,"num_read_kb":48767,"num_write":116,"num_write_kb":33370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"114'14","reported_seq":89,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:49.785203+0000","last_change":"2024-04-24T16:01:49.785177+0000","last_active":"2024-04-24T16:01:49.785203+0000","last_peered":"2024-04-24T16:01:49.785203+0000","last_clean":"2024-04-24T16:01:49.785203+0000","last_became_active":"2024-04-24T16:00:57.874746+0000","last_became_peered":"2024-04-24T16:00:57.874746+0000","last_unstale":"2024-04-24T16:01:49.785203+0000","last_undegraded":"2024-04-24T16:01:49.785203+0000","last_fullsized":"2024-04-24T16:01:49.785203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-04-24T16:01:49.785087+0000","last_deep_scrub":"114'14","last_deep_scrub_stamp":"2024-04-24T16:01:49.785087+0000","last_clean_scrub_stamp":"2024-04-24T16:01:49.785087+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:27:05.640816+0000","scrub_duration":0.010908764,"objects_trimmed":0,"snaptrim_duration":0.00087475200000000002,"stat_sum":{"num_bytes":4194496,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":8,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16777484,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"114'18","reported_seq":212,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T16:01:42.748267+0000","last_change":"2024-04-24T16:01:42.748231+0000","last_active":"2024-04-24T16:01:42.748267+0000","last_peered":"2024-04-24T16:01:42.748267+0000","last_clean":"2024-04-24T16:01:42.748267+0000","last_became_active":"2024-04-24T16:00:22.364071+0000","last_became_peered":"2024-04-24T16:00:22.364071+0000","last_unstale":"2024-04-24T16:01:42.748267+0000","last_undegraded":"2024-04-24T16:01:42.748267+0000","last_fullsized":"2024-04-24T16:01:42.748267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'18","last_scrub_stamp":"2024-04-24T16:01:42.748068+0000","last_deep_scrub":"114'18","last_deep_scrub_stamp":"2024-04-24T16:01:42.748068+0000","last_clean_scrub_stamp":"2024-04-24T16:01:42.748068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:13.918648+0000","scrub_duration":0.019820012000000001,"objects_trimmed":0,"snaptrim_duration":0.00069742100000000004,"stat_sum":{"num_bytes":8388739,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":5,"num_write":13,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":46,"num_read":7581,"num_read_kb":173537,"num_write":932,"num_write_kb":139905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165862,"num_keys_recovered":12,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1177,"ondisk_log_size":1177,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":83886673,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":125,"num_read_kb":4809,"num_write":70,"num_write_kb":131225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104857959,"num_keys_recovered":24,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":268,"num_omap_keys":28,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167870464,"data_stored":167773346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323284,"kb_used_data":21836,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94048556,"statfs":{"total":96636764160,"available":96305721344,"internally_reserved":0,"allocated":22360064,"data_stored":21544055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46500000000000002,"15min":0.46300000000000002},"min":{"1min":0.307,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46600000000000003,"15min":0.45600000000000002},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.46500000000000002,"15min":0.45000000000000001},"min":{"1min":0.34300000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.502}]},{"osd":2,"last update":"Wed Apr 24 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.47299999999999998},"min":{"1min":0.34699999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.623},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.375,"15min":0.37},"min":{"1min":0.253,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.378},"min":{"1min":0.28499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.622,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.67700000000000005}]},{"osd":4,"last update":"Wed Apr 24 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58499999999999996,"5min":0.622,"15min":0.622},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.749}]}]},{"osd":4,"up_from":11,"seq":47244640300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335560,"kb_used_data":34112,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036280,"statfs":{"total":96636764160,"available":96293150720,"internally_reserved":0,"allocated":34930688,"data_stored":34126895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71299999999999997,"5min":0.747,"15min":0.747},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.52200000000000002},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.5019999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.498},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.504}]},{"osd":2,"last update":"Wed Apr 24 16:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.317,"15min":0.317},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79000000000000004,"5min":1.877,"15min":1.877},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.69999999999999996,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.377}]},{"osd":5,"last update":"Wed Apr 24 16:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.41199999999999998},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.46899999999999997},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.67000000000000004,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40000000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640301,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327964,"kb_used_data":30612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94043876,"statfs":{"total":96636764160,"available":96300929024,"internally_reserved":0,"allocated":31346688,"data_stored":30523151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.69199999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.504},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.47099999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.57499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.623}]},{"osd":1,"last update":"Wed Apr 24 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.36699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57199999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59399999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63900000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65200000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64400000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40400000000000003,"15min":0.41199999999999998},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63100000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.67500000000000004}]},{"osd":5,"last update":"Wed Apr 24 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39900000000000002,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42899999999999999},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.70799999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323864,"kb_used_data":22416,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94047976,"statfs":{"total":96636764160,"available":96305127424,"internally_reserved":0,"allocated":22953984,"data_stored":22134482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52300000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.34399999999999997}]},{"osd":1,"last update":"Wed Apr 24 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61199999999999999,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38600000000000001,"15min":0.39000000000000001},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.32300000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.502,"15min":0.497},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75800000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Wed Apr 24 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.40500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54500000000000004}]}]},{"osd":0,"up_from":11,"seq":47244640301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335576,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94036264,"statfs":{"total":96636764160,"available":96293134336,"internally_reserved":0,"allocated":34947072,"data_stored":34127087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.27500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.46000000000000002,"5min":0.504,"15min":0.504},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Apr 24 16:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.35799999999999998,"15min":0.36199999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.436,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42099999999999999,"15min":0.42399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.49299999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.35299999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68700000000000006,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54300000000000004,"15min":0.56000000000000005},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70899999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.66000000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72799999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53400000000000003,"15min":0.54300000000000004},"min":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.621}]},{"osd":5,"last update":"Wed Apr 24 16:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56000000000000005,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68700000000000006}]}]},{"osd":1,"up_from":11,"seq":47244640301,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331464,"kb_used_data":30016,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040376,"statfs":{"total":96636764160,"available":96297345024,"internally_reserved":0,"allocated":30736384,"data_stored":29932532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:01:45 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56000000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67400000000000004,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Apr 24 16:01:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34100000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38500000000000001},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57299999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Apr 24 16:01:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72399999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:01:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70999999999999996,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.77400000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:01:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.746,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.75700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33579008,"data_stored":33554740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":20971767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29384704,"data_stored":29360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:02:01.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T16:02:01.915 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T16:02:01.915 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T16:02:01.915 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T16:02:01.915 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:01.957 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T16:02:01.957 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T16:02:01.958 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:01.993 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T16:02:01.993 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T16:02:01.993 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.060 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T16:02:02.060 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T16:02:02.060 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.097 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T16:02:02.097 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T16:02:02.097 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.171 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T16:02:02.171 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T16:02:02.171 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.238 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T16:02:02.239 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T16:02:02.239 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T16:02:02.239 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.305 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T16:02:02.305 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T16:02:02.305 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T16:02:02.305 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.318 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T16:02:02.318 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T16:02:02.319 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.372 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T16:02:02.373 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T16:02:02.373 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T16:02:02.384 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T16:02:02.384 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T16:02:02.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1 2024-04-24T16:02:02.455 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T16:02:02.456 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T16:02:02.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T16:02:02.705 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T16:02:02.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T16:02:02.799 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T16:02:02.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T16:02:02.887 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T16:02:03.039 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T16:02:03.040 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T16:02:03.127 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T16:02:03.127 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T16:02:03.202 INFO:tasks.ceph:Archiving mon data... 2024-04-24T16:02:03.203 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/data/mon.a.tgz 2024-04-24T16:02:03.204 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-24T16:02:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.QVtrCQ6NOU 2024-04-24T16:02:03.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QVtrCQ6NOU 2024-04-24T16:02:03.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.QVtrCQ6NOU 2024-04-24T16:02:03.461 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.QVtrCQ6NOU is 269KB 2024-04-24T16:02:03.534 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.QVtrCQ6NOU 2024-04-24T16:02:03.556 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/data/mon.c.tgz 2024-04-24T16:02:03.556 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-24T16:02:03.617 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.10mLqlxOLu 2024-04-24T16:02:03.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.10mLqlxOLu 2024-04-24T16:02:03.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.10mLqlxOLu 2024-04-24T16:02:03.871 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.10mLqlxOLu is 389KB 2024-04-24T16:02:03.968 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.10mLqlxOLu 2024-04-24T16:02:03.989 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/data/mon.b.tgz 2024-04-24T16:02:03.990 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-04-24T16:02:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.iijRzXJDeP 2024-04-24T16:02:04.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.iijRzXJDeP 2024-04-24T16:02:04.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.iijRzXJDeP 2024-04-24T16:02:04.258 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.iijRzXJDeP is 389KB 2024-04-24T16:02:04.354 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.iijRzXJDeP 2024-04-24T16:02:04.370 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T16:02:04.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T16:02:04.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T16:02:04.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T16:02:04.446 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-24T16:02:04.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T16:02:04.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T16:02:04.477 DEBUG:teuthology.orchestra.run.smithi122:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T16:02:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T16:02:04.503 INFO:teuthology.orchestra.run.smithi086.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T16:02:04.520 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T16:02:04.528 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T16:02:04.530 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi032/crash 2024-04-24T16:02:04.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T16:02:04.567 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi086/crash 2024-04-24T16:02:04.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T16:02:04.594 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi122/crash 2024-04-24T16:02:04.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T16:02:04.653 INFO:tasks.ceph:Compressing logs... 2024-04-24T16:02:04.653 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T16:02:04.656 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T16:02:04.658 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T16:02:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T16:02:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-04-24T16:02:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41738.log 2024-04-24T16:02:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2024-04-24T16:02:04.683 INFO:teuthology.orchestra.run.smithi086.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-04-24T16:02:04.685 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41738.log: /var/log/ceph/ceph-client.admin.41689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41738.log.gz 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41831.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41831.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T16:02:04.686 INFO:teuthology.orchestra.run.smithi086.stderr: 83.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T16:02:04.689 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46428.log 2024-04-24T16:02:04.689 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57387.log 2024-04-24T16:02:04.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61932.log 2024-04-24T16:02:04.689 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46428.log.gz 2024-04-24T16:02:04.698 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-04-24T16:02:04.698 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41783.log 2024-04-24T16:02:04.698 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-04-24T16:02:04.698 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41783.log.gz 2024-04-24T16:02:04.745 INFO:teuthology.orchestra.run.smithi086.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T16:02:04.814 INFO:teuthology.orchestra.run.smithi086.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T16:02:04.826 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T16:02:04.843 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61516.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57387.log.gz 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59353.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61932.log.gz 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61403.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61516.log.gz 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59353.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.59353.log.gz--verbose 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:-- /var/log/ceph/ceph-client.admin.52670.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65367.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61403.log.gz 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46681.log 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52670.log.gz 2024-04-24T16:02:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64607.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65367.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46681.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46681.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.62088.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60708.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64607.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57901.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62088.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61297.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60708.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57901.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57901.log.gz-5 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:--verbose -- /var/log/ceph/ceph-client.admin.54727.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61297.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61297.log.gz 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54250.log 2024-04-24T16:02:04.845 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54727.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66447.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54250.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65987.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63440.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66447.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65987.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53508.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65320.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63440.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48285.log 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53508.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65320.log.gz 2024-04-24T16:02:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58403.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65550.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48285.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48216.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58403.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50570.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65550.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47759.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48216.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58136.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50570.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46516.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47759.log.gz 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64140.log 2024-04-24T16:02:04.847 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58136.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46516.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64653.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46957.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64140.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66079.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64653.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59061.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46957.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53210.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66079.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59061.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61426.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62404.log 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53210.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61426.log.gz 2024-04-24T16:02:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56329.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61940.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62404.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64469.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56329.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56928.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61940.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64469.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56928.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47105.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46521.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62616.log 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47105.log.gz 2024-04-24T16:02:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46521.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58253.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66769.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62616.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53553.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58253.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59799.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66769.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53553.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56211.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45905.log 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59799.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56211.log.gz 2024-04-24T16:02:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66033.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46241.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45905.log.gz 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57295.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66033.log.gz 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66401.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46241.log.gz 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57295.log.gz 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55890.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49368.log 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66401.log.gz 2024-04-24T16:02:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65318.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55890.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49368.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.49368.log.gz--verbose 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:-- /var/log/ceph/ceph-client.admin.46522.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52528.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65318.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53861.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62405.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52528.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53861.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48992.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55470.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62405.log.gz 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46412.log 2024-04-24T16:02:04.852 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48992.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65282.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55470.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46412.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57318.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65282.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46704.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63847.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57318.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46704.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52402.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63847.log.gz 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56598.log 2024-04-24T16:02:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54500.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52402.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56598.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54500.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.54500.log.gz--verbose 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:-- /var/log/ceph/ceph-client.admin.51169.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51169.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66263.log 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-04-24T16:02:04.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56553.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66263.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49508.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56553.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65681.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63662.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49508.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65681.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59354.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63662.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65684.log 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gz 2024-04-24T16:02:04.855 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59354.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59354.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65684.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.65684.log.gz-- 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51873.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57119.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63954.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51873.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48124.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57119.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47049.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63954.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr: 91.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65680.log 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48124.log.gz 2024-04-24T16:02:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47049.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57775.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65895.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65680.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57775.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63731.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46849.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65895.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58277.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63731.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46849.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58277.log.gz 2024-04-24T16:02:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55221.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.55221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55221.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45857.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45857.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62473.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.50922.log: -- /var/log/ceph/ceph-client.admin.66355.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46021.log 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62473.log.gz 2024-04-24T16:02:04.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59532.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66355.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45951.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46021.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54451.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59532.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45951.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54451.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59693.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50791.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63171.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59693.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62851.log 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50791.log.gz 2024-04-24T16:02:04.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63171.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62851.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49687.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49687.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61864.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65086.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61864.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65086.log.gz 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59648.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51605.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52917.log 2024-04-24T16:02:04.860 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59648.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51605.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52917.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64515.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48078.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46243.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64515.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48078.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46243.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60283.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66217.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60283.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37167.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60283.log.gz 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-04-24T16:02:04.861 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66217.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.37167.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37167.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46075.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.46075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46075.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54679.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66493.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54679.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66493.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51941.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65803.log 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51941.log.gz 2024-04-24T16:02:04.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65803.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64234.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47197.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66125.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64234.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61647.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47197.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66125.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61647.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60389.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65941.log 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60389.log.gz 2024-04-24T16:02:04.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65941.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51406.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57143.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51406.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.57143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57143.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63170.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59602.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63170.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59602.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56873.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47313.log 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56873.log.gz 2024-04-24T16:02:04.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47313.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48262.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57752.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57752.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60093.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47243.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60093.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47243.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64423.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65085.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64423.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65085.log.gz 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50428.log 2024-04-24T16:02:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50428.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55846.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51112.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55846.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51112.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50006.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53143.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50006.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53143.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63532.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55094.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63532.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55094.log.gz 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52083.log 2024-04-24T16:02:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64280.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52083.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64280.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66171.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50063.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66171.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50063.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66815.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52824.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66815.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52824.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50346.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65366.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50346.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65366.log.gz 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60788.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65088.log 2024-04-24T16:02:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60788.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65088.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58674.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48020.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58674.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48020.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54925.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66631.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54925.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66631.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62806.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62298.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62806.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46728.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55341.log 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62298.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46728.log.gz 2024-04-24T16:02:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55341.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49731.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49731.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46240.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61027.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46240.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61027.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57842.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57842.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64791.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64791.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2024-04-24T16:02:04.869 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55561.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66585.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55561.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66585.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64929.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46844.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64929.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64331.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46844.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64699.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64331.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64699.log.gz 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64883.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46242.log 2024-04-24T16:02:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64883.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46242.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54796.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53211.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54796.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53211.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66869.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60013.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66869.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60013.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58522.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61814.log 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58522.log.gz 2024-04-24T16:02:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61814.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63125.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63125.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58768.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54476.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58768.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54476.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49263.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49263.log.gz 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48352.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60481.log 2024-04-24T16:02:04.872 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48352.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60481.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64561.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47429.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64561.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47429.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61084.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65826.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61084.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65826.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64377.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64745.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64377.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64745.log.gz 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58815.log 2024-04-24T16:02:04.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58815.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65087.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48170.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65087.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48170.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48673.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65597.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48673.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50747.log 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65597.log.gz 2024-04-24T16:02:04.874 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50747.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51674.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64837.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51674.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64837.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49049.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66309.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49049.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66309.log.gz 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56154.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58207.log 2024-04-24T16:02:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56154.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58207.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48717.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58859.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48717.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58859.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54155.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63639.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63639.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62231.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64095.log 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62231.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64095.log.gz 2024-04-24T16:02:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66539.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64187.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66539.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64187.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65266.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47003.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65266.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65090.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46792.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65090.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46792.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46519.log 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46519.log.gz 2024-04-24T16:02:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57431.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65527.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57431.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65527.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66723.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66677.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66723.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66677.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65089.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65089.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63274.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58042.log 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63274.log.gz 2024-04-24T16:02:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58042.log.gz 2024-04-24T16:02:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51803.log 2024-04-24T16:02:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66913.log 2024-04-24T16:02:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51803.log.gz 2024-04-24T16:02:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66913.log.gz 2024-04-24T16:02:04.879 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T16:02:04.880 INFO:teuthology.orchestra.run.smithi086.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T16:02:04.887 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39107.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37571.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37854.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39107.log.gz 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37824.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37571.log.gz 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37854.log.gz 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-04-24T16:02:04.888 INFO:teuthology.orchestra.run.smithi122.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.37824.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41635.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38850.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41635.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 18.1% 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38850.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-04-24T16:02:04.889 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37907.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37907.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38094.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr: 56.7% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39941.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-04-24T16:02:04.890 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38094.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39941.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37021.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz--verbose 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:-- /var/log/ceph/ceph-client.admin.38751.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40262.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.38751.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40262.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-04-24T16:02:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41531.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41531.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39307.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39307.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38372.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39696.log 2024-04-24T16:02:04.892 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38969.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.39696.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38598.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.38969.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38598.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37514.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37514.log.gz 2024-04-24T16:02:04.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38088.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37251.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38088.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39865.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37251.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38711.log 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz 2024-04-24T16:02:04.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38711.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41167.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41167.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39790.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40524.log 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-04-24T16:02:04.895 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39756.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39790.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.40524.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40198.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40342.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38848.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40342.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38368.log 2024-04-24T16:02:04.896 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40648.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38848.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40200.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38368.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41324.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40648.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40200.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39512.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41324.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41298.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39512.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38811.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41298.log.gz 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 2024-04-24T16:02:04.897 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39533.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38811.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38458.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39533.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38168.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38458.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38146.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38168.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38146.log.gz 2024-04-24T16:02:04.898 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37933.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37999.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 51.4% -- replaced with /var/log/ceph/ceph-client.admin.37933.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38569.log 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-04-24T16:02:04.899 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 52.4% -- replaced with /var/log/ceph/ceph-client.admin.37999.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38569.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38231.log 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-04-24T16:02:04.900 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38656.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr: 54.8% -- replaced with /var/log/ceph/ceph-client.admin.38231.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38656.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.38656.log.gz-- 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39445.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-04-24T16:02:04.901 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37698.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37689.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 4.8% -- replaced with /var/log/ceph/ceph-client.admin.37698.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37689.log: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.37689.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39619.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38539.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40563.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39619.log.gz 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39765.log 2024-04-24T16:02:04.902 INFO:teuthology.orchestra.run.smithi122.stderr: 51.7% -- replaced with /var/log/ceph/ceph-client.admin.38539.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40563.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39765.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr: 61.4% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr: 45.1% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-04-24T16:02:04.903 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40574.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38900.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37815.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38900.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38960.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.37815.log.gz 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38960.log: gzip -5 --verbose -- 18.1%/var/log/ceph/ceph-client.admin.40410.log 2024-04-24T16:02:04.904 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38960.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37802.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38528.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39434.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37802.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37779.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38528.log.gz 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-04-24T16:02:04.905 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39434.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37779.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37962.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37962.log.gz 2024-04-24T16:02:04.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38058.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.38058.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-04-24T16:02:04.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41593.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-04-24T16:02:04.908 INFO:teuthology.orchestra.run.smithi122.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38391.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41593.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40468.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.38391.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40468.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38742.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40923.log 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr: 27.0% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-04-24T16:02:04.909 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.38742.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.40923.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39474.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz/var/log/ceph/ceph-client.admin.39068.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39068.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37672.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40990.log 2024-04-24T16:02:04.910 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37672.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41655.log: 52.7% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40479.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38261.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40479.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38261.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-04-24T16:02:04.911 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.41176.log -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41236.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38026.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41176.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41236.log: 51.6% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38026.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40175.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40043.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40043.log.gz 2024-04-24T16:02:04.912 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37724.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37865.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.37724.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.37865.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37113.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40703.log 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-04-24T16:02:04.913 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40429.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37113.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.40703.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40427.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.40429.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39550.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37903.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40427.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40239.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.39550.log.gz 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40652.log 2024-04-24T16:02:04.914 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37903.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip /var/log/ceph/ceph-client.admin.40239.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40239.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40652.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40047.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39161.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40047.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39161.log.gz 2024-04-24T16:02:04.915 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37531.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40822.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37531.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.40822.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39290.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 41.5% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40438.log 2024-04-24T16:02:04.916 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39290.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38702.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.40438.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38702.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38030.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-04-24T16:02:04.917 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38030.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38511.log 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr: 34.2% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38511.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38479.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz-5 2024-04-24T16:02:04.918 INFO:teuthology.orchestra.run.smithi122.stderr:--verbose -- /var/log/ceph/ceph-client.admin.39905.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41090.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39905.log.gz 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38047.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.41090.log.gz 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38195.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38047.log.gz 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36929.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz/var/log/ceph/ceph-client.admin.39010.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-04-24T16:02:04.919 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38195.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39010.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37596.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz-- 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41676.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38740.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38740.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40701.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39726.log 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40701.log.gz 2024-04-24T16:02:04.920 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39726.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 54.1% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40533.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38781.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 45.0% -- replaced with /var/log/ceph/ceph-client.admin.40533.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40856.log 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr: 57.5% -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-04-24T16:02:04.921 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38781.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.40856.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38430.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38217.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39628.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38430.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38217.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39628.log.gz 2024-04-24T16:02:04.922 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38437.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38437.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38594.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38615.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40317.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38594.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38617.log 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38615.log.gz 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40317.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.40317.log.gz--verbose 2024-04-24T16:02:04.923 INFO:teuthology.orchestra.run.smithi122.stderr:-- /var/log/ceph/ceph-client.admin.39489.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38617.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.38617.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr: 51.5% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-04-24T16:02:04.924 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39245.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.39245.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37874.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36975.log 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37874.log: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.37874.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36975.log.gz 2024-04-24T16:02:04.925 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-24T16:02:04.926 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m0.089s 2024-04-24T16:02:04.926 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m0.095s 2024-04-24T16:02:04.926 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.107s 2024-04-24T16:02:04.926 INFO:teuthology.orchestra.run.smithi032.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T16:02:04.932 INFO:teuthology.orchestra.run.smithi032.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T16:02:04.960 INFO:teuthology.orchestra.run.smithi032.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T16:02:04.983 INFO:teuthology.orchestra.run.smithi086.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T16:02:04.985 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-04-24T16:02:04.985 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.313s 2024-04-24T16:02:04.985 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.776s 2024-04-24T16:02:04.985 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.034s 2024-04-24T16:02:05.288 INFO:teuthology.orchestra.run.smithi032.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T16:02:05.289 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-24T16:02:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m0.615s 2024-04-24T16:02:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m1.353s 2024-04-24T16:02:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.196s 2024-04-24T16:02:05.289 INFO:tasks.ceph:Archiving logs... 2024-04-24T16:02:05.290 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi032/log 2024-04-24T16:02:05.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T16:02:05.741 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi086/log 2024-04-24T16:02:05.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T16:02:05.884 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi122/log 2024-04-24T16:02:05.885 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T16:02:06.136 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T16:02:06.163 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T16:02:06.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T16:02:06.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T16:02:06.168 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T16:02:06.274 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T16:02:06.284 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T16:02:06.284 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:02:06.287 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:02:06.290 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:02:06.305 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-24T16:02:06.306 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-04-24T16:02:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:^- hv01.front.sepia.ceph.com 2 6 377 33 +1522us[+1511us] +/- 581ms 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -814us[ -825us] +/- 61ms 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1339us[+1328us] +/- 89ms 2024-04-24T16:02:06.310 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:02:06.311 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:02:06.311 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-04-24T16:02:06.312 INFO:teuthology.orchestra.run.smithi086.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +1497us[+1497us] +/- 581ms 2024-04-24T16:02:06.312 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -838us[ -835us] +/- 61ms 2024-04-24T16:02:06.312 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1321us[+1321us] +/- 89ms 2024-04-24T16:02:06.312 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:02:06.312 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:02:06.313 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-24T16:02:06.313 INFO:teuthology.orchestra.run.smithi122.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +1494us[+1487us] +/- 581ms 2024-04-24T16:02:06.313 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -859us[ -866us] +/- 61ms 2024-04-24T16:02:06.313 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1302us[+1295us] +/- 89ms 2024-04-24T16:02:06.313 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:02:06.314 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T16:02:06.323 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T16:02:06.323 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T16:02:06.333 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T16:02:06.444 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T16:02:06.522 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T16:02:06.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:02:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:06.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:02:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:06.665 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:02:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:06.698 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T16:02:06.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T16:02:06.718 INFO:teuthology.task.internal:Duration was 1399.975447 seconds 2024-04-24T16:02:06.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T16:02:06.727 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T16:02:06.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T16:02:06.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T16:02:06.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T16:02:06.780 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:02:06.786 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:02:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:02:07.232 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T16:02:07.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T16:02:07.232 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T16:02:07.262 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-04-24T16:02:07.266 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T16:02:07.292 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-24T16:02:07.292 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T16:02:07.315 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T16:02:07.315 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T16:02:07.318 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T16:02:07.335 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T16:02:07.400 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T16:02:07.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T16:02:07.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T16:02:07.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T16:02:07.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T16:02:07.513 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T16:02:07.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T16:02:07.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T16:02:07.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T16:02:07.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T16:02:07.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.640 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-04-24T16:02:07.643 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-24T16:02:07.665 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-24T16:02:07.679 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:07.730 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:07.746 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:02:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:02:07.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T16:02:07.773 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T16:02:07.774 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi032 2024-04-24T16:02:07.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T16:02:07.822 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi086 2024-04-24T16:02:07.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T16:02:07.862 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671585/remote/smithi122 2024-04-24T16:02:07.862 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T16:02:07.900 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T16:02:07.901 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T16:02:07.904 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T16:02:07.906 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T16:02:07.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T16:02:07.963 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T16:02:07.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T16:02:07.973 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T16:02:07.973 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T16:02:07.976 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T16:02:07.978 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T16:02:07.992 INFO:teuthology.orchestra.run.smithi086.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 16:02 /home/ubuntu/cephtest 2024-04-24T16:02:07.995 INFO:teuthology.orchestra.run.smithi032.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 16:02 /home/ubuntu/cephtest 2024-04-24T16:02:08.009 INFO:teuthology.orchestra.run.smithi122.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 16:02 /home/ubuntu/cephtest 2024-04-24T16:02:08.010 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T16:02:08.021 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T16:02:08.044 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} thrashers/cache thrashosds-health workloads/rbd_nbd} duration: 1399.975447177887 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-24T16:02:08.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T16:02:08.122 INFO:teuthology.run:pass